The Mystery of Matthew

Doing a bit more on my family tree. These are my Welsh great-great-grandparents who married in 1863. I know that Matthew was born in 1839 and it seems that his mother died in childbirth or shortly after :-( I know where he was in 1861, aged 22, but I can’t find him in the 1841 or 1851 censuses. He and his wife Sarah also disappear for a bit in 1891 and 1901. A mystery that I haven’t been able to solve online. I think it will need the skill set and magic wand of my super sleuth archivist cousin.