LabellessFace: Fair Metric Learning for Face Recognition without Attribute Labels

Presented at IJCB2024

1. LabellessFace:
Tetsushi Ohki
1,2,*, Yuya Sato
1,*, Masakatsu Nishigaki
1, Koichi Ito
3

1Shizuoka University,
2RIKEN AIP,
3Tohoku University
Fair Metric Learning for Face Recognition
without Attribute Labels

2. Motivation
One of the most prominent concerns regarding racial bias is the
racial composition within the dataset. However, Constructing a fair
and fair and large-scale dataset is often have problems.
Human-annotated labels poses
challenges in terms of time, cost, and
potential biases.
Scalability Label dependency
Label dependent fairness improvement
cannot guarantee fairness for unknown
attributes
Can we improve a fairness notion without
assuming the target attribute labels?

3. LabellessFace | Framework
Normalized
feature
Fair class margin penaltyFace recognition
model
Normalized weights
Update class
favoritism level
Softmax
Cross-entropy loss
Each epoch
Weights
-Dynamically updates different
margins for each individuals based
on class favoritism level while
progressing the training through the
fair class margin penalty process
-Fair class margin penalty is defined
based on the class favoritism level
-Class favoritism level is determined
based on how much the recognition
accuracy for each individual deviates
from the overall average using the
training samples.

4. Normalized
feature
Fair class margin penaltyFace recognition
model
Normalized weights
Update class
favoritism level
Softmax
Cross-entropy loss
Each epoch
Weights
LabellessFace | Fair Class Margin Penalty
margin
coefficient
class
favoritism level
A coefficient dc (margin coefficient) is added to the
basic ArcFace loss function to minimize the bias in
individual authentication accuracy.

5. LabellessFace | Class Favoritism Level
<latexit sha1_base64="OvMG1a6seQaHfJwt8OM/agCa86M=">AAADhHicjVLNThRBEK5lFHH94e9C4mXDZo0Hs6lFEcLBkHDxuIALJCxuZsbepcP8pWd2ASfwALwAJp4g8WB8DC+8AAcewXDExIsHvu4ZNLpB6MlMV31dX1V90+VEnowT5rPCgHXn7uC9ofvFBw8fPR4eGR1bicOuckXDDb1QrTl2LDwZiEYiE0+sRUrYvuOJVWdrQZ+v9oSKZRi8TXYjseHbnUC2pWsngN41Qxxqblpvyb3WSJmrbFap36jlRpnyVQ9HC0xNek8hudQlnwQFlMD2yKYYzzrViCkCtkEpMAVLmnNBe1QEt4sogQgb6Ba+HXjrORrA1zljw3ZRxcOrwCxRhU/5C1/wCX/l7/zr2lypyaF72cXuZFwRtYYPJpZ/3sjysSe0+YcFRuVajgCulakbtCXUplmjSUJjZBCt1s3q9D4cXizPLVXSp3zM59B5xGf8DUqD3g/386JY+vSfvh30nFUP4G+bv+YbHQHuKQWub6FjkB1EauRKZYhs2ldASnnc/u/IJrRJeBKx8S1raH23q5FF9tcoYiZr/05gv7EyVa29qk4vvizPT+XTOURPaJKeYQJnaJ7eUJ0a6FTRRzqiY2vQem69sKaz0IFCzhmnv5b1+hKhZMW/</latexit>
Pi
average
average
average
<latexit sha1_base64="C2SNkiRsrtwuhWLsFwxdkSIt6p4=">AAADeXicjVK7bhNBFD328gjmkQQaJBqDZRRRWNfhKapINJRJjJNISWTtLmNnlH1pd20IFnwAEi0pqECiQHwGDT9AkU9AlEGigIIzswsIrJDManfuPXPPvffsXC8JdJaL7FWqzrHjJ05OnaqdPnP23PTM7PmVLB6mvur6cRCna56bqUBHqpvrPFBrSarc0AvUqrd9z5yvjlSa6Th6kO8kajN0B5Hua9/NCXX6Pd2baUhL7KpPGu3SaKBci/FsRbCBh4jhY4gQChFy2gFcZHzW0YYgIbaJMbGUlrbnCk9RI3fIKMUIl+g2vwN66yUa0Tc5M8v2WSXgm5JZR1M+yTvZl4/yXj7LjwNzjW0O08sOd6/gqqQ3/fxi59uhrJB7jq0/LDKaB3IUcaMsPURbjj7uWE2aGhOLGLV+UWf0ZHe/c3e5Ob4qb+QLdb6WPflApdHoq/92SS2/+k/fHnsuqkf0H9m/FlodEe9pTNzcwsAijxlpkF8qY2YzfkqkXsY9+x25QW2anmZsdsQaRt/RahSRkzVqnMn2vxM4aazMt9q3WjeXbjQW5svpnMIlXMEcJ/A2FnAfi+iy0wFe4CV2q9+dy86cc60IrVZKzgX8tZzrPwG9McGu</latexit>
fi
<latexit sha1_base64="Cves9VRMJMxHWvJ0Ime/9hk+vjM=">AAADeXicjVK7bhNBFD328gjmkQQaJBqDZRRRWNfhKapINJRJjJNISWTtLmMzZF/aXRuCBR+AREsKKpAoEJ9Bww9Q5BMQZZAooODM7AICKySz2p17z9xz7z0710sCneUiu5Wqc+ToseNTJ2onT50+Mz0ze3Yli4epr7p+HMTpmudmKtCR6uY6D9Rakio39AK16m3dMeerI5VmOo7u5duJ2gzdQaT72ndzQp1+72FvpiEtsas+abRLo4FyLcazFcEG7iOGjyFCKETIaQdwkfFZRxuChNgmxsRSWtqeKzxFjdwhoxQjXKJb/A7orZdoRN/kzCzbZ5WAb0pmHU35JO9kTz7Ke/ksP/bNNbY5TC/b3L2Cq5Le9PPznW8HskLuOR78YZHR3JejiBtl6QHacvRxy2rS1JhYxKj1izqjJzt7ndvLzfFleSNfqPO17MoHKo1GX/23S2r51X/69thzUT2i/8j+tdDqiHhPY+LmFgYWecxIg/xSGTOb8VMi9TLu2e/IDWrT9DRjs0PWMPoOV6OInKxR40y2/53ASWNlvtW+0bq+dK2xMF9O5xQu4BLmOIE3sYC7WESXnQ7wAi+xU/3uXHTmnCtFaLVScs7hr+Vc/QnARMGv</latexit>
fj
Face Recognition
Model
softmax
<latexit sha1_base64="0bR206Aly3H3HxRwtifYC8NJX70=">AAAEGXicjVLLbtNAFL2ugZbwaAobJDYjolSpENEkghYhkCqxYdkHaSt1qmg8GTuj+sV4Ehqs8AH8AIuuisQC8RWITcUWsegnIJZFYsOCa8c8o5KOZc+9Z86ZO8dzndhXiaH0yJqyz5w9Nz1zvnTh4qXLs+W5KxtJ1NNCtkTkR3rL4Yn0VShbRhlfbsVa8sDx5aaz+zBb3+xLnagofGwGsdwJuBcqVwluEGqXDzpt8YD50jUsJcyRngpTrjUfDFPfH5ISczUXaXOYNshNwuReXGMeDwJOmOhEhrhtsTAk86SGAblP6AJhbJKm+6e2kDJPPsnVMuwU9ZlWXtfUS+1yhdZpPsh40CiCChRjJZqzKDDoQAQCehCAhBAMxj5wSPDZhgZQiBHbgRQxjZHK1yUMoYTaHrIkMjiiu/j1MNsu0BDzbM8kVwus4uOrUUmgSj/RN/SYHtK39DP9fuJeab5HdpYBzs5IK+P27Itr698mqgKcDXR/q1BRPVEjEc+c6QneDLhwN/ek0GOcI5lbMarTf/byeP3eWjWdp6/oF/R5QI/oe3Qa9r+K16tybf8/53bwzKPqIeZP878W5D5CvKcU8ewWvBzZQ2aG/HQZ4W5ZrhEhBe/5LyZDbwozhdzklDUyf6erMWKO18h6svFvB44HG816Y7F+Z/V2ZblZdOcMXIcbUMMOXIJleAQr0AJhTVu3rEVryd6339mH9ocRdcoqNFfhr2F//AH0SvgO</latexit>
dc=
(
2
1+exp(!·fc)
(fc<0)
2
1+exp(!·h·fc)
(fc!0)
<latexit sha1_base64="U1WeUPg/qUgtdkUwCCpi5WW4+Bs=">AAADjnicjVLLThRBFL1D+4DxwaAJIXEzYTLqalJDzGhMjAQ2LHk4QMKQSXdbDAX9SlXNCLbDB/gDLlhBwoLgX7jxB1zwCYYlJm5ceKq61egEoTrdde+pe+69p+t6SSCUZuy0MORcu37j5vBI8dbtO3dHS2P3llXclT5v+nEQy1XPVTwQEW9qoQO+mkjuhl7AV7ztWXO+0uNSiTh6pXcTvh66nUhsCN/VgNql8ZbmOzptKi7L/a3MeaT67VKF1Zhd5UGjnhsVytd8PFZg1KLXFJNPXQqJU0QadkAuKTxrVCdGCbB1SoFJWMKec+pTEdwuojgiXKDb+HbgreVoBN/kVJbto0qAV4JZpir7wo7ZOfvMTthX9uPCXKnNYXrZxe5lXJ60R99PLH2/lBVi17T5hwVG9UIOB26UyUu0adqgZ1aTgMbEIkatn9Xpvf1wvvR8sZo+ZIfsDDoP2Cn7BKVR75t/tMAX9//Tt4ees+oR/Df2r4VWR4R7SoGbW+hYZAeRBvmlMkY240sg5Txu73dkC9oEPIFYdcUaRt/VamSRgzWKmMn6vxM4aCxP1eqNWmPhSWV6Jp/OYXpAk/QYE/iUpmmO5qmJTt/RIZ3QR6fkNJwXzsssdKiQc+7TX8uZ+wloMMl2</latexit>
Userj’s
<latexit sha1_base64="sLyPTHtj9BenwRjH0ailztBTgFo=">AAADjnicjVLNbtNAEJ7U/JQUaAoSQuISNQriFE0QCggJUcGlx/6lrdRUkW22YVV7ba03ocWUB+AFOHCiEgdU3oILL8Chj4A4FokLB75dGxBEpV3L3plv55uZzztBGsnMMB9WJrwzZ8+dn7xQnbp46fJ0bebKapYMdSi6YRIlej3wMxFJJbpGmkisp1r4cRCJtWD7sT1fGwmdyUStmN1UbMb+QMktGfoGUL92rWfEjslXtC+VVIO6xLnI9vq1BrfYrfq40S6NBpVrIZmpMPXoCSUU0pBiEqTIwI7IpwzPBrWJKQW2STkwDUu6c0F7VAV3iCiBCB/oNr4DeBslquDbnJljh6gS4dVg1qnJn/k9H/EnPuAv/OPYXLnLYXvZxR4UXJH2p19dX/5+IivGbujpHxYYzWM5ArhVpk/QZmiL7jlNEhpTh1i1YVFn9Pz10fL9pWZ+k/f5K3S+5UP+CKVq9C18tyiW3vyn7wA9F9UV/Gfur8VOh8I95cDtLQwcsoNIi/xSmSCb9TWQehn38ndkD9okPInY7JQ1rL7T1Sgix2tUMZPtfydw3Fi93Wp3Wp3FO425R+V0TtINmqVbmMC7NEfztEBddPqC9umAPng1r+M98B4WoROVknOV/lre/E9JTMls</latexit>
Training images
<latexit sha1_base64="sLyPTHtj9BenwRjH0ailztBTgFo=">AAADjnicjVLNbtNAEJ7U/JQUaAoSQuISNQriFE0QCggJUcGlx/6lrdRUkW22YVV7ba03ocWUB+AFOHCiEgdU3oILL8Chj4A4FokLB75dGxBEpV3L3plv55uZzztBGsnMMB9WJrwzZ8+dn7xQnbp46fJ0bebKapYMdSi6YRIlej3wMxFJJbpGmkisp1r4cRCJtWD7sT1fGwmdyUStmN1UbMb+QMktGfoGUL92rWfEjslXtC+VVIO6xLnI9vq1BrfYrfq40S6NBpVrIZmpMPXoCSUU0pBiEqTIwI7IpwzPBrWJKQW2STkwDUu6c0F7VAV3iCiBCB/oNr4DeBslquDbnJljh6gS4dVg1qnJn/k9H/EnPuAv/OPYXLnLYXvZxR4UXJH2p19dX/5+IivGbujpHxYYzWM5ArhVpk/QZmiL7jlNEhpTh1i1YVFn9Pz10fL9pWZ+k/f5K3S+5UP+CKVq9C18tyiW3vyn7wA9F9UV/Gfur8VOh8I95cDtLQwcsoNIi/xSmSCb9TWQehn38ndkD9okPInY7JQ1rL7T1Sgix2tUMZPtfydw3Fi93Wp3Wp3FO425R+V0TtINmqVbmMC7NEfztEBddPqC9umAPng1r+M98B4WoROVknOV/lre/E9JTMls</latexit>
Training images
softmax
<latexit sha1_base64="U/RhVGUaagrUBGjZ8Y3gJxySC1s=">AAADhHicjVLLThRBFL1DA+II8nBj4mbCZAwLMrnDQw0LQ+LG5fAYIGFg0t0WY0m/Ut0zgB38AH9AE1eQuDB+Bht/wAWfYFxi4saFp6objEx4VKe77j11z733dF0n8mScMJ8W+qz+gcE7Q3eL94ZH7o+OjU+sxWFHuaLhhl6oNhw7Fp4MRCORiSc2IiVs3/HEurP7Qp+vd4WKZRisJgeR2PLtdiB3pGsngLabIQ41N6233hy2xspcZbNKvUYtN8qUr3o4XmBq0isKyaUO+SQooAS2RzbFeDapRkwRsC1KgSlY0pwLOqQiuB1ECUTYQHfxbcPbzNEAvs4ZG7aLKh5eBWaJKvydv/AZf+Ov/IP/XJkrNTl0LwfYnYwrotbo+4crv29k+dgTev2PBUblSo4ArpWpG7QltEPPjCYJjZFBtFo3q9N9++FsZWG5kj7mY/4JnUd8yidQGnR/uZ+XxPKna/p20HNWPYC/Z/6ab3QEuKcUuL6FtkH2EamRc5UhsmlfASnlce8uIpvQJuFJxMa3rKH13a5GFtlbo4iZrF2ewF5jbaZae1KdX5orL87k0zlEj2iSpjCBT2mRXlKdGuhU0Uc6omNr0Jq2Zq35LLSvkHMe0H/Lev4XpHjFwA==</latexit>
Pj
class favoritism level
<latexit sha1_base64="GugXQiHLNm3c3I8oPNShaU3pcoo=">AAADjnicjVJLbhNBEC1n+ATziQMSQmJjYRlYWWWEDEJCRLDJMh+cRIoja2bomFbmp+62SRjMAbgAC1ZEYoHCLdhwARY5AmIZJDYseN0zgMAKSY9muup1vap60xVkkdSGeb8y5Z04eer09Jnq2XPnL8zUZi+u6HSoQtEN0yhVa4GvRSQT0TXSRGItU8KPg0isBluP7PnqSCgt0+Sx2cnERuwPErkpQ98A6tcu94zYNnlXC1Ufy8K5ocf9WoNb7FZ90miXRoPKtZDOVph69IRSCmlIMQlKyMCOyCeNZ53axJQB26AcmIIl3bmgMVXBHSJKIMIHuoXvAN56iSbwbU7t2CGqRHgVmHVq8md+zwf8iff4C/84NFfucthedrAHBVdk/ZlXV5a/H8mKsRt6+ocFRvNQjgBulakjtBnapLtOk4TGzCFWbVjUGT1/fbB8b6mZX+dd/gqdb3mfP0JpMvoWvlsUS2/+03eAnovqCfxn7q/FTkeCe8qB21sYOGQbkRb5pTJFNusrIPUy7uXvyB60SXgSsfqYNay+49UoIidrVDGT7X8ncNJYudVqd1qdxduNuYfldE7TVbpGNzGBd2iO5mmBuuj0Be3SHn3wal7Hu+89KEKnKiXnEv21vPmfZRTJdQ==</latexit>
Useri’s
<latexit sha1_base64="tt/u+dtqDJaPOPsT4zRHSdKGv5M=">AAADoHicjVI9TxRRFL3LqOD6wYKNic2EzRoaN3eNAUJFpKFzF1wgsmQzMz6WF+YrM7OrMFl+AIWthRUkFoSWFhsb/4AFP8FYYmJj4XlvRlE3CG8y8+49955735l37dCVccJ8Whgyrl2/MTxys3jr9p27o6Wx8eU46EaOaDqBG0SrthULV/qimcjEFathJCzPdsWKvTWv4is9EcUy8J8n26FY96yOLzekYyWA2qXJVoCwYqf1fnvefGSeA3+G+u1Smauslzlo1HKjTPmqB2MFpha9pIAc6pJHgnxKYLtkUYxnjWrEFAJbpxRYBEvquKA+FcHtIksgwwK6hW8H3lqO+vBVzVizHXRx8UZgmlThz3zIZ/yJj/gL/7iwVqprqLNsY7czrgjbo3v3l75fyvKwJ7R5zgKjciFHAFfKoku0JbRBM1qThMZQI0qtk/Xp7bw9W5pdrKQP+YC/Quc+n/JHKPV735z3DbH47j/ntnHmrLsP/5X+a57W4eOeUuDqFjoaeY1MhfxSGaCa8iMgZp63+zuzBW0SnkRufMUeSt/VemSZgz2KmMnavxM4aCw/rtamqlONJ+W5p/l0jtADmqBJTOA0zdEC1amJk76hYzqhD8aEsWA8MxpZ6lAh59yjv5bx4icb/tEM</latexit>
PC!P
<latexit sha1_base64="qXX4CoVSFrrvA5/K5//xUbcnysk=">AAADj3icjVLLTttAFL3BfdD0QUIXrdRN1ChVV9FNVSXAAkV0Q3fhEUAiKLLdIYzwS7aTQq3wAfwAi65o1UXV9iu66Q90wSegLqnEhkXPjE1RhVIYy/a9Z86ZO2fmWoEjo5j5KDdm3Lh56/b4nfzde/cfTBSKkyuR3w9t0bZ9xw/XLDMSjvREO5axI9aCUJiu5YhVa/uVml8diDCSvrcc7wZiwzV7ntyUthkD6hYedXxMK3VyEbWGw26hzNXp6To3pkqXg1qV9ShTNlp+McfUoTfkk019ckmQRzFih0yK8KxTjZgCYBuUAAsRST0vaEh5aPtgCTBMoNv49pCtZ6iHXK0ZabWNKg7eEMoSVfgnf+YT/sFf+JjPRq6V6DXUXnbxt1KtCLoT+4+XTq9UufjHtHWhgqIyUiOAK2fhFd5i2qQp7UnCY6AR5dZO6wzeHZwszSxWkmf8gX/B5yEf8Xc49Qa/7U8LYvH9f/ZtYc9pdQ/5W31qrvbh4Z4S4OoWehrZAVMh5y59rKbyEEgp4+39ZXbgTSKT4EbXrKH8Xa9GyrxcI4+ePG+80uhg5UW1Vq/WF16Wm3NZd47TE3pKz9GBDWrSPLWorU/4I32lb0bRaBizRjOljuUyzUP6Zxiv/wBqxsrA</latexit>
P
<latexit sha1_base64="415X51qSNvuF78/R0QNbK7MeRKI=">AAADeXicjVI9bxNBEH328RHMRxJokGgMllFEYY0BBUQVQUOZxDiJlETW3bF2Vrkv3Z0NwYIfgERLCiqQKBA/g4Y/QJGfgCiDRAEFb/cOEFgh2dPdzrydNzPvdrwk0FkuslepOseOnzg5dap2+szZc9Mzs+dXsniY+qrrx0GcrnlupgIdqW6u80CtJalyQy9Qq972PXO+OlJppuPoQb6TqM3QHUS6r303J9Tp9/zeTENaYld90miXRgPlWoxnK4INPEQMH0OEUIiQ0w7gIuOzjjYECbFNjImltLQ9V3iKGrlDRilGuES3+R3QWy/RiL7JmVm2zyoB35TMOprySd7JvnyU9/JZfhyYa2xzmF52uHsFVyW96ecXO98OZYXcc2z9YZHRPJCjiBtl6SHacvRx22rS1JhYxKj1izqjJ7v7nTvLzfFVeSNfqPO17MkHKo1GX/23S2r51X/69thzUT2i/8j+tdDqiHhPY+LmFgYWecxIg/xSGTOb8VMi9TLu2e/IDWrT9DRjsyPWMPqOVqOInKxR40y2/53ASWPleqs935pfutlYuFtO5xQu4QrmOIG3sID7WESXnQ7wAi+xW/3uXHbmnGtFaLVSci7gr+Xc+Amv4cG5</latexit>
fc
<latexit sha1_base64="HDoCsqqHr2EFbC1lHRoDNPQThdg=">AAACuHichVFLLwNRFP46XvVssZHYCCq1aW5FkK4aNpYtqhKamhlXXeZlZtqoiT/gD1jYILEQe3/ARmJt4SeIZT02Fs5MmwgNzuTOPfc75zv3O/coliYcl7HHkNTS2tbeEe7s6u7p7YtE+wdWHbNsqzynmppprymywzVh8JwrXI2vWTaXdUXjeWVvwY/nK9x2hGmsuFWLF3S5ZIhtocouQYVMfEPRvYOj4u5kcbcYHWMJFthIs5NsOGPp8deb20r3W8aM3mMDWzChogwdHAZc8jXIcOhbRxIMFmEFeITZ5IkgznGELuKWKYtThkzoHv1LdFpvoAad/ZpOwFbpFo2WTcwRxNgDu2I1dseu2RP7+LWWF9TwtVRpV+pcbhUjx0PL7/+ydNpd7Hyx/tTsYhtzgVZB2q0A8btQ6/zK4UltObUU8ybYBXsm/efskd1SB0blRb3M8qVTqh77VREn3H83+4+X86hD3Y/TIJM/x9bsrE4lkjOJ6SxNdB51C2MYo4jT3GaRxiIyyFH9fZzgDOdSStqUSpKop0qhBmcQ30yyPwFvPaGg</latexit>
P(xj)j
<latexit sha1_base64="tBDIEPeY5CXTtrU34EKEl22tqAg=">AAACuHichVE7SwNBEP48X/EdtRFsxBCJCGFPRMUqaGMZHzGCCfHuXOPivby7BOORP2AtWNioYCH+DBvB2iI/QSwVbCycuwREQ+Icezv7zXyz3+yoti5cj7Fah9TZ1d3TG+nrHxgcGh6Jjo7tuFbJ0XhGs3TL2VUVl+vC5BlPeDrftR2uGKrOs+rxWhDPlrnjCsvc9io2zxtK0RSHQlM8gvLpRE41/NNqQcwWRCEaY0kW2lSzIzecWGo6N3dRS1XSVvQZORzAgoYSDHCY8MjXocClbw8yGGzC8vAJc8gTYZyjin7iliiLU4ZC6DH9i3Taa6AmnYOabsjW6BadlkPMKcTZC7tn7+yJPbBX9tWylh/WCLRUaFfrXG4XRs4ntj7/ZRm0ezj6YbXV7OEQy6FWQdrtEAm60Or88tnl+9bKZtyfYbfsjfTfsBp7pA7M8od2t8E3r6h6vKUiTnjwbk6bl/OpQyOI0yDlv2Nrdnbmk/JicmGDJrqKukUwiWkkaG5LSGEdaWSo/gkucY0baUXal4qSqKdKHQ3OOH6Z5HwD0wygZA==</latexit>
P(xi)i
<latexit sha1_base64="G64al792m6qC4AyNwF4JuoyaU3k=">AAAC4XichVHPS9xQEP421tZarVutUOhFKis9lOVFpBVBEHvxuK6uu2BsSF6fGje/SN4u6iN/gJ5KoQjuSaGH4j8h9GLbswf/BOlRwUsPnWQXiop2QjLzvplv8s0bO3SdWDJ2ltO6HnQ/fNTzuPdJX//TgfyzwaU4aERcVHjgBlHNtmLhOr6oSEe6ohZGwvJsV1Tt+vs0X22KKHYCf1FuhWLFs9Z8Z9XhliTIzE8YniXXueWqWmJuTBvKsD21mZhq4009McjXp/Xkg7pWlZj5UVZkmY3cDvROMDoz9ryafKkOl4L8Lxj4iAAcDXgQ8CEpdmEhpmcZOhhCwlagCIsocrK8QIJe4jaoSlCFRWidvmt0Wu6gPp3TnnHG5vQXl96ImCMosFP2jV2wE3bEztmfO3uprEeqZYu83eaK0BzYfbFw9V+WR15i/R/rXs0Sq5jMtDqkPcyQdAre5je39y4WpsoFNcYO2W/Sf8DO2HeawG9e8q/zotyi7oU7FQnC03uL7rk5RRN6aZ4Wqd9c2+1gabyovy1OzNNGZ9G2HrzEK7ymvb3DDOZQQoX67+MYP/BT49qO9kn73C7Vch3OEK6Z1voLlHyxkw==</latexit>
Xj={xj,k}
Xj
k=1
<latexit sha1_base64="+zrhFHCeRbPnDSAODLSqUbGd4MQ=">AAAC4XichVFNSxxBEH1OTGI0xtUYCHiRyIoHWXpEogiC6CVHv9ZdcMww07ba7Hwx07tEm/kB8SQBCcSTQg6SPxHwkujZgz9BcjSQSw6pmV0IKmoNM1X9ql7Nqy438mSiGLtoMx61P37ytONZZ9fz7hc9hd6+lSSsx1yUeeiFcdV1EuHJQJSVVJ6oRrFwfNcTFbc2l+UrDREnMgyW1XYk1nxnM5AbkjuKILswbvmO2uKOp6upLactbbm+/pDaWo7WUot8bdpM3+trValdGGIlltvg7cBsBUMzw/2VdL/yaj4snMHCOkJw1OFDIICi2IODhJ5VmGCICFuDJiymSOZ5gRSdxK1TlaAKh9AafTfptNpCAzpnPZOczekvHr0xMQdRZOfsmF2xH+wbu2R/7+yl8x6Zlm3ybpMrIrtn9/XSnwdZPnmFrf+sezUrbGAy1ypJe5Qj2RS8yW/sfL5amlos6mF2xH6R/kN2wU5ogqDxm39dEIsH1L14pyJBeHZv8T03p2lCP8vTIs2ba7sdrIyVzLel8QXa6Cya1oEBvMEI7W0CM3iHeZSp/xd8x0+cGtz4aOwZn5qlRluL8xLXzDj4B41hsZA=</latexit>
Xi={xi,k}
Xi
k=1
Fair class
margin
penalty
Average
confidence score of
each individual
Overall average
confidence score
Set of
confidence scores

6. Margin coefficient
<latexit sha1_base64="Y8zmh9SJvkO0I8eEqm/56+c8DHo=">AAADeXicjVI9bxNBEH328RHMRxJokGgMllFEYY0hShBVBA1lEuMkUhJZd5eNWeW+dHc2BAt+ABItKahAokD8DBr+AEV+AqIMEgUUvN07QGCFZE93O/N23sy82/GSQGe5yH6l6pw4eer0xJna2XPnL0xOTV9cyeJB6quuHwdxuua5mQp0pLq5zgO1lqTKDb1ArXo798z56lClmY6jB/luojZDtx/pbe27OaHOVs/vTTWkJXbVx412aTRQrsV4uiLYwBZi+BgghEKEnHYAFxmfdbQhSIhtYkQspaXtucJT1MgdMEoxwiW6w2+f3nqJRvRNzsyyfVYJ+KZk1tGUT/JODuSjvJfP8uPQXCObw/Syy90ruCrpTT6/3Pl2JCvknuPhHxYZzUM5irhRlh6hLcc2bltNmhoTixi1flFn+GTvoHNnuTm6Lm/kC3W+ln35QKXR8Kv/dkktv/pP3x57LqpH9B/ZvxZaHRHvaUTc3ELfIo8ZaZBfKmNmM35KpF7GPfsduUFtmp5mbHbMGkbf8WoUkeM1apzJ9r8TOG6s3Gy151pzS7ONhbvldE7gCq5hhhM4jwXcxyK67LSPF3iJvep356oz49woQquVknMJfy3n1k+pt8G3</latexit>
dc
ℎ=1
ℎ=0
ℎ=0.3
visualization of
<latexit sha1_base64="Y8zmh9SJvkO0I8eEqm/56+c8DHo=">AAADeXicjVI9bxNBEH328RHMRxJokGgMllFEYY0hShBVBA1lEuMkUhJZd5eNWeW+dHc2BAt+ABItKahAokD8DBr+AEV+AqIMEgUUvN07QGCFZE93O/N23sy82/GSQGe5yH6l6pw4eer0xJna2XPnL0xOTV9cyeJB6quuHwdxuua5mQp0pLq5zgO1lqTKDb1ArXo798z56lClmY6jB/luojZDtx/pbe27OaHOVs/vTTWkJXbVx412aTRQrsV4uiLYwBZi+BgghEKEnHYAFxmfdbQhSIhtYkQspaXtucJT1MgdMEoxwiW6w2+f3nqJRvRNzsyyfVYJ+KZk1tGUT/JODuSjvJfP8uPQXCObw/Syy90ruCrpTT6/3Pl2JCvknuPhHxYZzUM5irhRlh6hLcc2bltNmhoTixi1flFn+GTvoHNnuTm6Lm/kC3W+ln35QKXR8Kv/dkktv/pP3x57LqpH9B/ZvxZaHRHvaUTc3ELfIo8ZaZBfKmNmM35KpF7GPfsduUFtmp5mbHbMGkbf8WoUkeM1apzJ9r8TOG6s3Gy151pzS7ONhbvldE7gCq5hhhM4jwXcxyK67LSPF3iJvep356oz49woQquVknMJfy3n1k+pt8G3</latexit>
dc
<latexit sha1_base64="415X51qSNvuF78/R0QNbK7MeRKI=">AAADeXicjVI9bxNBEH328RHMRxJokGgMllFEYY0BBUQVQUOZxDiJlETW3bF2Vrkv3Z0NwYIfgERLCiqQKBA/g4Y/QJGfgCiDRAEFb/cOEFgh2dPdzrydNzPvdrwk0FkuslepOseOnzg5dap2+szZc9Mzs+dXsniY+qrrx0GcrnlupgIdqW6u80CtJalyQy9Qq972PXO+OlJppuPoQb6TqM3QHUS6r303J9Tp9/zeTENaYld90miXRgPlWoxnK4INPEQMH0OEUIiQ0w7gIuOzjjYECbFNjImltLQ9V3iKGrlDRilGuES3+R3QWy/RiL7JmVm2zyoB35TMOprySd7JvnyU9/JZfhyYa2xzmF52uHsFVyW96ecXO98OZYXcc2z9YZHRPJCjiBtl6SHacvRx22rS1JhYxKj1izqjJ7v7nTvLzfFVeSNfqPO17MkHKo1GX/23S2r51X/69thzUT2i/8j+tdDqiHhPY+LmFgYWecxIg/xSGTOb8VMi9TLu2e/IDWrT9DRjsyPWMPqOVqOInKxR40y2/53ASWPleqs935pfutlYuFtO5xQu4QrmOIG3sID7WESXnQ7wAi+xW/3uXHbmnGtFaLVSci7gr+Xc+Amv4cG5</latexit>
fc
and (γ=10)
‣The margin coefficient dc is designed to
increase the margin's impact on classes
that are less favored (fc < 0)
‣Enlarging the facial feature space, while
reducing the margin's impact on classes
that are more favored (fc ≧ 0)
‣Parameter h adjusts the influence on the
more favored classes.
‣Parameter γ adjusts the sensitivity of the
output to the input.
less
favored
more
favored
Margin coefficient dc can be get from output of a function similar to the reversed sigmoid
function that takes Class Favoritism Level (fc)

7. Evaluation | Objectives
(2) Label-Independent Fairness Improvement
(1) Fairness-Accuracy Trade-off
Can our proposed method improve the fairness of certain sensitive attributes
(e.g. race or gender) while maintaining accuracy?
In our experiment, we evaluate the effectiveness of our proposed
LabellessFace framework by addressing the following two main questions.
Can our proposed method improve fairness independent of annotated labels?

8. Evaluation | Protocol
[9] Wang, Mei, and Weihong Deng. "Mitigating bias in face recognition using skewness-aware reinforcement learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[10] Wang, Mei, et al. "Racial faces in the wild: Reducing racial bias by information maximization adaptation network." Proceedings of the ieee/cvf international conference on computer vision. 2019.
[11] Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller. Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments. University of Massachusetts, Amherst, Technical Report 07-49.
Dataset
‣RFW for Fairness-Accuracy Trade-off Evaluation
‣LFW for Label-Independent Fairness Improvement Evaluation
Model ResNet-34
MetricEER, AUC (Performance)
STD, Gini, SER (Fairness)
BUPT-Balancedface[9] (train), RFW[10] (test), LFW[11] (test)

9. Results | Fairness-Accuracy Trade-off
-CIFP achieved lower EER since it utilize a training algorithm that takes into
account pre-labeled racial information
-Proposed method exhibits better fairness performance (STD/Gini/SER) with
relatively lower EER indicating that proposed method improves fairness while
maintaining authentication performance
EER-Af(↓)EER-As(↑)EER-Ca(↑)EER-In(↑)STD(↓) Gini(↓) SER(↓)
ArcFace 0.1847 0.1975 0.1145 0.1621 0.03163 0.01031 0.1725
MagFace 0.2034 0.1905 0.0989 0.154 0.04054 0.1353 2.056
CIFP 0.1683 0.173 0.097 0.1293 0.03097 0.1175 1.78
MixFairFace 0.4661 0.2869 0.2928 0.3155 0.07349 0.1032 1.627
Proposed 0.181 0.1871 0.1163 0.1625 0.02775 0.08922 1.609
✓Evaluated EER (performance)
✓How much difference is there in EER across racial attributes? (STD/Gini/SER) (fairness)
✓Compare with ArcFace, MagFace, CIFP and MixFairFace that take different races into account.
Af: African / As: Asian / Ca: Caucasian / In: Indian

10. Results | Label-Independent Fairness Improvement
-The proposed method performs best on all
performance and fairness metrics.
-Label-free fairness training at an individual level
can achieve a high trade-off between accuracy
and fairness even for unknown attributes.
'BJSOFTT)FBUNBQPG1SPQPTFE.FUIPE
EER(↓)AUC(↑)STD(↓)Gini(↓)SER(↓)
ArcFace 0.0930.96650.01170.08292 2.766
MagFace0.09867 0.9590.011270.08279 2.766
CIFP 0.0910.96140.011570.08845 3.038
Proposed 0.0910.96810.010190.07398 2.525
✓We selected 26 attributes that have more than 100 samples from LFW Dataset
✓How much difference is there in EER across various unknown attributes? (fairness)
✓Compare proposed method with ArcFace, MagFace and CIFP.

11. Discussions
How should we define the hyperparameters? (e.g. γ or h?)
Isn't the class favoritism level computationally expensive?
We can compute the class favoritism level sequentially during training, which
results in only a small increase in time and spatial computational complexity.
However, since the computation increases in proportion to the number of training
data, this aspect needs to be considered.
We set the hyper parameters γ=10 and h=1 through grid search to balance the
trade-off between fairness and accuracy. Larger value of h or γ could lead to
greater fairness improvement in case there exists significant latent attribute
biases.

12. Conclusion
✓The proposed LabellessFace framework aims to achieve fairness in
facial recognition without requiring demographic group labeling.
✓It introduces a "fair class margin penalty" based on class favoritism
levels, eliminating the need for demographic labels to ensure fairness.
✓Experiments on facial benchmarks demonstrated the method's
effectiveness in achieving fairness compared to other baselines,
without requiring fairness considerations during training.
✓We open up new possibilities for creating more equitable and scalable
face recognition systems
project site