Ok this question has haunted my brain for a bit and it is very annoying so let's see if I can figure this out
For some reason I am unable to conceive of the proper way to add angular momenta in a complex system of bodies which both rotate and orbit each other but I think I figured it out also I think better when I think "out loud"
Ok so the moon is a sphere (ish) so I_moon approx = 2/5 m_moon r_moon^2 (from here when i say = i mean approx =)
With rotation speed w_moon, its angular momentum in its own frame is I_moon w_moon
But from the center of the earth, we need to use the parallel axis theorem to get I_moon orbit = I_moon + m_moon d_moon^2. Calculating its angular momentum from this perspective we get L_moon orbit = I_moon w_month + m_moon d_moon^2 w_month. But then this implicitly assumes that the moon is at the end of a "rod" effectively attaching it to the center of the earth, which then necessitates that the moon is rotating with w_moon = w_month.
Substitute w_moon for w_month on the "moon is rotating about its own axis" part of that angular momentum we get L_moon orbit = I_moon w_moon + m_moon d_moon^2 w_month. That makes sense. That also implies angular momentum adds in a very logical way. Which is something I could not find any confirmation on. It felt too simple, therefore it felt wrong.
Since angular momentum adds up nice and linearly, we can now add the Earth's angular momentum to the earth-moon system. Earth has I_earth = 2/5 m_earth r_earth^2, and contributes I_earth w_day angular momentum for a total of L_earth moon system = I_earth w_day + I_moon w_moon + m_moon d_moon^2 w_month.
Due to tidal forces, the earth and moon are slowly becoming tidally locked. Once that happens, w_day = w_moon = w_month = w_final, and L_earth moon system = (I_earth + I_moon + m_moon d_moon final^2) w_final.
Angular momentum is conserved, so:
(I_earth + I_moon + m_moon d_moon final^2) w_final = I_earth w_day + I_moon w_moon + m_moon d_moon^2 w_moon
We have one equation and two unknowns (d_moon final and w_final). Tidal forces do result in a loss of some of that energy to heat, but I'm not certain how to estimate that currently so we will assume no heat is produced and all energy remains in the motion of the system.
No heat is produced:
E_initial = E_final
E_initial = 1/2 (I_earth w_day^2 + I_moon w_moon^2 + m_moon d_moon^2 w_month^2)
E_final = 1/2 (I_earth + I_moon + m_moon d_moon final^2) w_final^2
Now we have two equations with two unknowns. Now for some hard numbers:
m_earth = 5.972168 e24 kg
r_earth = 6.3710 e6 m
m_moon = 7.342 e22 kg
r_moon = 1.7374 e6 m
d_moon = 3.84399 e8 m
w_earth = 2 pi / (23hr 56min 4.100sec) = 7.292 e-5 Hz
w_moon = 2 pi / (27.321661day) = 2.6616995 e-6 Hz
w_month = 2 pi / (27.321661day) = 2.6616995 e-6 Hz
I_earth = 0.3307 m_earth r_earth^2 = 8.016 e37 kg m^2 [from Wikipedia MOI factor]
I_moon = 0.3929 m_moon r_moon^2 = 8.708 e34 kg m^2 [from Wikipedia MOI factor]
m_moon d_moon^2 = 1.08487 e40 kg m^2
And now we can find initial angular momentum and kinetic energy
L_initial = I_earth w_day + I_moon w_moon + m_moon d_moon^2 w_moon = 5.845 e33 Js + 2.318 e29 Js + 2.8876 e34 Js = 3.4721 e34 Js
E_initial = 1/2 (I_earth w_day^2 + I_moon w_moon^2 + m_moon d_moon^2 w_month^2) = 2.131 e29 J + 3.084654 e23 J + 3.843 e28 J = 2.5153 e29 J
Moon's rotation is kinda negligible but that's fair. Earth dominates kinetic energy but moon orbit dominates angular momentum. Now to solve the pair of equations:
(I_earth + I_moon + m_moon d_moon final^2) w_final = 3.4721 e34 Js
1/2 (I_earth + I_moon + m_moon d_moon final^2) w_final^2 = 2.5153 e29 J
This isn't too bad. We recognize I_earth + I_moon + m_moon d_moon final^2 = I, consistent across both equations. Then
I w_final = 3.4721 e34 Js
1/2 I w_final^2 = 2.5153 e29 J
w_final = 3.4721 e34 Js / I -> w_final^2 = 1.2055 e69 J^2 s^2 / I^2
w_final^2 = 5.0306 e29 J / I
1.2055 e69 J^2 s^2 / I^2 = 5.0306 e29 J / I
I 1.2055 e69 J^2 s^2 = I^2 5.0306 e29 J
I = 1.2055 e69 J^2 s^2 / (5.0306 e29 J) = 2.3963 e39 kg m^2
We can solve this out for d_moon final to get
m_moon d_moon final^2 = I - I_earth - I_moon = 2.3161 e39 kg m^2
d_moon final^2 = 3.1545 e16 m^2
d_moon final = 1.7761 e8 m, or actually significantly closer to the earth than it is presently. I believe I have done a mistake somewhere as it should be getting further away. The mistake has to be in the fact that the moon's orbital angular momentum is dominating the earth's rotational angular momentum. It's past midnight so maybe I've done something quite stupid in here.
Whatever, let's find that tidally locked day.
w_final = 3.4721 e34 Js / I = 1.4489 e-5 Hz = 2 pi / day length
day length = 5 days 27 min
See that makes sense. It makes sense that the earth's days will get longer. The earth is actively slowing its rotation due to tidal friction with the moon, which orbits the earth slower than the earth rotates. What doesn't make sense is the moon getting closer. Or maybe it does make sense? For the earth to slow down, the moon needs to speed up. If its orbit drifts further away... it would slow down. So it would need to get closer. But I critically assumed that energy would be conserved, no losses to heat. So that would quite possibly account for the discrepancy with observed reality that the moon is drifting away from us.
How account for energy losses due to friction 😭 why are my physics skills so rusty 😭