Sometimes when you have nodes or other entities listed in your sitemap.xml, the lastmod element may not be accurate when it is just based on the node.changed property. changefreq and priority are also essentially useless, as Google has been ignoring them for a while.
There is an open issue about adding settings to disable these. In the meantime, it's pretty easy to remove them:
/**
* Implements hook_xmlsitemap_element_alter().
*
* @param array $element
* @param array $link
* @param \Drupal\xmlsitemap\XmlSitemapInterface $sitemap
*/
function MYMODULE_xmlsitemap_element_alter(array &$element, array $link, XmlSitemapInterface $sitemap) {
unset($element['changefreq'], $element['lastmod'], $element['priority']);
}
Add this to a custom module (I always have a generic "SITENAME_system" module for things like this), and regenerate your sitemap.