JavaScript (ES6), 112 bytes

All credit for this shorter version goes to @nwellnhof.

Takes input as an integer. Returns an array of three decimal digits.

n=>[(x=n>>4,y=x>>3,q=n/2&55,p=q%8)>5&&q-39?8|y&1:y,(p&5^5?x&6:q-23?8:y&6)|x&1,(p<5?p*2:p<6?x&6:p%q<7?y&6:8)|n&1]

Try it online!

JavaScript (ES6), 118 117 bytes

Takes input as an integer. Returns an array of three decimal digits.

n=>[(x=n>>4&7,y=n>>7,p=n/2&7)>5&&p<7|x/2^2?8|y&1:y,(p<7?p-5?x:8:x/2^1?8:y&6)|x&1,(p<5?p*2:p<6?x&6:p<7|x<2?y&6:8)|n&1]

Try it online!

How?

Instead of trying to apply the 'official' algorithm, this code is based on some kind of reverse-engineering of the patterns that can be found in the expected results.

Given the input integer $n$, we compute:

$$begin{align}&x=leftlfloorfrac{n}{16}rightrfloor bmod 8\
&y=leftlfloorfrac{n}{128}rightrfloor\
&p=leftlfloorfrac{n}{2}rightrfloor bmod 8end{align}
$$

Example: first digit (hundreds)

x     | 0                | 1                | 2                | 3               

n & 1 | 0101010101010101 | 0101010101010101 | 0101010101010101 | 0101010101010101

p     | 0011223344556677 | 0011223344556677 | 0011223344556677 | 0011223344556677

------+------------------+------------------+------------------+-----------------

y = 0 | 0000000000008888 | 0000000000008888 | 0000000000008888 | 0000000000008888

y = 1 | 1111111111119999 | 1111111111119999 | 1111111111119999 | 1111111111119999

y = 2 | 2222222222228888 | 2222222222228888 | 2222222222228888 | 2222222222228888

y = 3 | 3333333333339999 | 3333333333339999 | 3333333333339999 | 3333333333339999

y = 4 | 4444444444448888 | 4444444444448888 | 4444444444448888 | 4444444444448888

y = 5 | 5555555555559999 | 5555555555559999 | 5555555555559999 | 5555555555559999

y = 6 | 6666666666668888 | 6666666666668888 | 6666666666668888 | 6666666666668888

y = 7 | 7777777777779999 | 7777777777779999 | 7777777777779999 | 7777777777779999



x     | 4                | 5                | 6                | 7               

n & 1 | 0101010101010101 | 0101010101010101 | 0101010101010101 | 0101010101010101

p     | 0011223344556677 | 0011223344556677 | 0011223344556677 | 0011223344556677

------+------------------+------------------+------------------+-----------------

y = 0 | 0000000000008800 | 0000000000008800 | 0000000000008888 | 0000000000008888

y = 1 | 1111111111119911 | 1111111111119911 | 1111111111119999 | 1111111111119999

y = 2 | 2222222222228822 | 2222222222228822 | 2222222222228888 | 2222222222228888

y = 3 | 3333333333339933 | 3333333333339933 | 3333333333339999 | 3333333333339999

y = 4 | 4444444444448844 | 4444444444448844 | 4444444444448888 | 4444444444448888

y = 5 | 5555555555559955 | 5555555555559955 | 5555555555559999 | 5555555555559999

y = 6 | 6666666666668866 | 6666666666668866 | 6666666666668888 | 6666666666668888

y = 7 | 7777777777779977 | 7777777777779977 | 7777777777779999 | 7777777777779999

Algorithm:

• If $p<6$, we have $d=y$
  


• If $p=6$, we have $d=8+(ybmod2)$
  


• If $p=7text{ AND }(x<4text{ OR }x>5)$, we have $d=8+(ybmod2)$
  


• If $p=7text{ AND }(x=4text{ OR }x=5)$, we have $d=y$
  

As JS code:

p > 5 && p < 7 | x / 2 ^ 2 ? 8 | y & 1 : y

Python 3, 229 ... 102 98 bytes

lambda a:[[a&6,a>>4&6,a>>7&6,8][b"yWeB7O]s"[a&8and(~a&6or a>>4&6|1)]%x&3]|a>>x%9&1for x in[7,4,9]]

Try it online!

-4 bytes by @xnor

-6 bytes by @nwellnhof

Formatted:

h = lambda a:[

    [a&6,a>>4&6,a>>7&6,8][                 List to take high bits from

        b"yWeB7O]s"[                       8 char string; where to get high bits for

                                             each of 8 possible indicator values

            a&8 and (~a&6 or a>>4&6|1)]    Compute indicator (by @nwellnhof)

        %x&3]                              High bits of each digit

    | a >> x%9 & 1                         bitwise OR with low bit of each digit

    for x in [7,4,9]]

Explanation

Because I originally wanted to implement this in Jelly, I take a different approach from most answers here, which is simple and perhaps suited to a golfing language. Though the golfed function takes an integer, let the input as a bit list be [a0,a1,...,a9]. Then we can derive three values from the input

• The low bits [a2,a5,a9]: These will always be the low bits of [d0,d1,d2] respectively.


• The high bits [2*a0a1,2*a3a4,2*a7a8,8]: The high bits of each digit will be one of these.


• The indicator bits, [a3,a4,a5,a7,a8], determining how to get the high bits of each digit. We compute the indicator (between 0 and 7) as follows:
  
  
  
  
  • If a5 == 0, the indicator is 0
  
  
  • If a3 nand a4, the indicator is 6 - 2*a3a4
  
  
  • Otherwise the indicator is 2*a7a8 + 1.
  
  
  
  

Then the nth digit can be elegantly computed as high_bits[arr[indicator][n]] | low_bits[n] by the table below, which is compressed into a string.

arr = [

    [0,1,2],

    [3,1,2],

    [1,3,2],

    [2,1,3],

    [2,3,3],

    [3,2,3],

    [3,3,2],

    [3,3,3]

]

JavaScript (Node.js), 126 119 117 112 111 bytes

(a,b,c,d,e,f,g,h,i,j)=>[(g&h&i+(b+=a*4+b,e+=d*4+e)!=5?8:b)+c,(g&i?h+e-3?8:b:e)+f,(g?h<i?e:h>i*e?b:8:h*4+i*2)+j]

Try it online!

-5 bytes thanks @tsh (and 2 by myself) So l can make more effort than I expected.

-2 more bytes using @tsh's technique!

-5 bytes thanks @Arnauld

-1 byte thanks @Neil

Input as a list of 10 bits (as 10 arguments), output as a list of 3 digits.

C (gcc), 138 129 bytes

f(w){int t=w/2&55,s=t%8,v=w/16,u=v/8;w=((s<6|t==39?u:8|u%2)*10+v%2+(s&5^5?v&6:t-23?8:u&6))*10+w%2+(s<5?s*2:s<6?v&6:s%t<7?u&6:8);}

Try it online!

First extracts some bits into variables s and t, so that the eight rows of the conversion table can be identified by:

1.  s < 4              u v w¹

2.  s = 4              u v 8¹

3.  s = 5              u 8 v

4.  s = 6              8 v u

5.  s = 7, t =  7      8 8 u

6.  s = 7, t = 23      8 u 8

7.  s = 7, t = 39      u 8 8

8.  s = 7, t = 55      8 8 8



¹ Can be computed with s*2

Then sets up u and v with divisions (right shifts), so that u, v and the input w contain the lower three BCD bits in positions 0-2. The rest is bit shuffling depending on s and t. Two notable tricks are:

s&5^5  // Rows 1, 2 and 4.

s%t<7  // Rows 1-5.

A port of Shieru Asakoto's Javascript solution is only 124 bytes:

f(a,b,c,d,e,f,g,h,i,j){a=(((g&h&i+(b+=a*4+b,e+=d*4+e)!=5?8:b)+c)*10+(g&i?h+e-3?8:b:e)+f)*10+(g?h-i|h&!e?h?b:e:8:h*4+i*2)+j;}

Try it online!

Ruby, 153 ... 119 117 bytes

->n{n+=n&896;a,b,c=n&1536,n&96,n&14;"%x"%n+=c<9?0:2036+[b/16-b-1918,r=a>>8,[r+126,a/16-26,a-1978,38][b/32]-a][c/2-5]}

Try it online!

How it works:

->n{n+=n&896;

This is the starting point: convert to BCD by shifting 3 bits to the left, which works for most of the patterns.

a,b,c=n&1536,n&96,n&14;

Get the middle bits of each nibble (and one extra bit of the third nibble, but mask the least significant bit).

"%x"%n+=c<9?0

If the third digit is less than 10 (less than 9 because we never cared for the LSB anyway), we're set: this is plain BCD, we can output the hex without changing anything

:2036+[b/16-b-1918,r=a>>8,[r+126,a/16-26,a-1978,38][b/32]-a][c/2-5]}

Otherwise do some black magic by shifting bits around and adding magic numbers until we get the result we want.

Retina 0.8.2, 191 181 bytes

(...)(...)

:$1,$2;

..(.),11(.);111

100$1,100$2;100

(10|(..)(.,)01)(.);111

100$3$2$4;100

(..)(.),(00.);111

100$2,1$3;0$1

(..)((.{5});110|(.);101)

100$3$4;$1

1

01

+`10

011

.0+(1*)

$.1

Try it online! Link includes test cases. Edit: Saved 10 byte by not padding digits to 4 bits except where necessary. Explanation:

(...)(...)

:$1,$2;

Insert separators so that each digit can be converted to decimal separately. This effectively handles the first two cases in the conversion table.

..(.),11(.);111

100$1,100$2;100

Handle the last (eighth) case in the conversion table.

(10|(..)(.,)01)(.);111

100$3$2$4;100

Handle the sixth and seventh cases in the conversion table.

(..)(.),(00.);111

100$2,1$3;0$1

Handle the fifth case in the conversion table.

(..)((.{5});110|(.);101)

100$3$4;$1

Handle the third and fourth cases in the conversion table.

1

01

+`10

011

.0+(1*)

$.1

Perform binary to decimal conversion.

Python 2, 157 bytes

lambda a,b,c,d,e,f,g,h,i,j:[c+[a*4+b*2,8][g*h*~(d*~e*i)],f+[d*4+e*2,8,a*4+b*2][g*i+(d<e)*g*i*h],j+[h*4+i*2,[8,[a*4+b*2,d*4+e*2][h<i]][h^i or(h&i-(d|e))]][g]]

Try it online!

Clean, 238 ... 189 bytes

-2 bytes thanks to Neil

import StdEnv

$a b c d e f g h i j=100*(c+2*b+4*a)+10*(f+2*e+4*d)+j+2*i+4*h-2*(h*(99*b+198*a-394)+i*(9*e+18*d+h*(e+2*d-4+(b+2*a-4)*(1-10*e-100*d+110*e*d))-35)-4)*g+0^(e+d)*(2*b+4*a-8*i*h*g)

Try it online!

Takes a 'list' of 10 bits in the form of 10 arguments, using a direct formula to compute the result.