https://groups.google.com/forum/#!topic/prometheus-users/0ZkYVj_8X8Q
First, for the snapshot when you launch prometheus, you'll need to add
--web.enable-admin-api
Then after that, you can trigger a snapshot by hitting:
curl -XPOST http://<promethues IP:Port>/api/v2/admin/tsdb/snapshot
This will respond with something like:
{"name":"2018-04-30T21:45:35Z-6e4d24be51aa32e3a"}
From the prometheus server, look in <value of --storage.tsdb.path>/snapshots and you'll see a dir with the name matching the response above.
If you need to recover from snapshot, you can just stop prometheus, delete the items in the storage.tsdb.path directory, and put the contents of the snapshot directory in its place. Or you can update the actual value of the storage.tsdb.path to point to the snapshot, but I prefer the first way.
The snapshot command can be ran from a cron job or similar to trigger regular cleanups.
Since it's creating hard links, space usage and time to initially create is minimal. But if you move the files, and over time you'll see the space needs increase.
You can just rm contents of the snapshot folder when they're no longer needed, or copy them off to a different location after the snapshot is generated The snapshot can not just target a different location initially.