AnnouncementsMatrixEventsFunnyVideosMusicAncapsTechnologyEconomicsPrivacyGIFSCringeAnarchyFilmPicsThemesIdeas4MatrixAskMatrixHelpTop Subs
3
Scrapist - 1 month ago - (+2|-0)
Today my exploits were far more conventional. But I thought I would share anyway.(self)
3 comments

save

byx0x7

It's a simple data need. Developing a library of historic price data. A lot of these APIs for historic prices will only let you get minutely data from the very recent past. So if you want a large collection of it in the future you need to set up a bot now.

Once again my interests are in scraping polymarket as they have been lately.

The only annoying bit is getting the ids for the markets that can't be fully automated. But I was able to get even that partially automated. I also adjusted the config files for it to require minimal manual formatting on my part.

The longest piece of code is the scrape script with 80% deals with reading the configs.

run.js 

#!/usr/bin/env node
module.exports = run;
async function run() {

var fs = require('fs');

var configFileNames = fs.readdirSync(__dirname+'/configs').filter(i=>i.endsWith('.config'));
//console.log({configFileNames});
var configs = [];
await Promise.all(configFileNames.map(async (filename)=>{
 var content = await fsp.readFile(__dirname+'/configs/'+filename,'utf8');
 content = content.split('\n').filter(i=>i);
 var [title,...marketLines] = content;
 //console.log({filename,content,title,marketLines});
 var savetitle = title.toLowerCase().split(' ').join('_');
 for(var marketLine of marketLines) {
  var [market,price] = marketLine.split(' ');
  configs.push({market,title,price,savetitle});
 }
}));
//console.log(configs);

//var URLPredicate = 'https://clob.polymarket.com/prices-history?market=';
//var URLSuffix = '&fidelity=10&interval=1m';
//https://clob.polymarket.com/prices-history?market=60803003979377314175239643215398210162298788643179995980032138901387043323508&fidelity=1&startTs=1737349200 200000

var URLPredicate = 'https://clob.polymarket.com/prices-history?market=';
var URLMiddle = '&fidelity=1&startTs=';
var time = Date.now();
var URLTime = Math.round((time-1000*60*60*24)/1000);
var URLSuffix = URLMiddle+URLTime;

await Promise.all(configs.map(async config=>{
 var url = URLPredicate+config.market+URLSuffix;
 var resp = await fetch(url);
 var data = await resp.json();
 var saveobj = {config,data,time};
 var {savetitle,price} = config;
 await fsp.writeFile(__dirname+'/data/'+savetitle+'.'+price+'.'+time+'.json',JSON.stringify(saveobj));
}));

};

This is what a config file looks like for this. I have a directory of 7 of them in total. The first line is a title and then it has pairs of market ids and the price for that sub-market.

** bitcoinjan.config ** 

Bitcoin January
60803003979377314175239643215398210162298788643179995980032138901387043323508 200000
33515038173925817579858963877943817944789731803907969874480156720410570347195 150000
38476885377229557440305324072246396823392362455910814239095246001075380350490 140000
77252770716032169427248210536603791716366460125006751242155179652569535874066 130000
73494102346489743269760351689951324890172428679575127088372315020339514789512 120000
9128895199999412805746501539402383368058380710575752735828347503045541925710 110000
55549274640733017542163520774439582975615803348529796706457461980106954184828 85000
80817664234388195477280224598403775170632677321946020704450003455110909930155 80000
41569140327535555873014778097123057668755541762133074730672257197323456023621 70000
112918611901220013725322222636611141155427593977261205533146706583406101029216 60000
80164953650010144148698886169109405470335795048037055067988523874664976610670 50000
78166787686154413342878841020173988743599281911730520933914808681047623872037 105000
92462795563289304883301490926149504610583825413905670008451532951671044015678 100000
85960683101891616445603305956751684944915110216625837134369504614420179134831 90000

To make it easy get the lower half of that file I have this one liner for the browser. 


console.log(Array.from($0.children).map(i=>i.id.replace('-checkbox','')+' '+i.innerText.replace(/[$,]/g,'')).join('\n'))

Last is a simple scheduler to run it once a day. I've abandoned cron for these things. I have a lot of JS bots that I've automated launching in tmux on boot.

** index.js ** 

#!/usr/bin/env node
process.chdir(__dirname);

const freq = 1000*60*60*24*1;
const run = require('./run');

function setIntervalDo(func,ms) {
 process.nextTick(func);
 return setInterval(func,ms)
}

setIntervalDo(async ()=>{
 console.log('Start get data');
 await run();
 console.log('Got data');
},freq);

Lastly I might as well share my script for launching all of my bots from the bot directory according to whatever is enabled in the quicklaunch file.

** launchbots.sh ** 

#!/bin/bash
# Ensure the quicklaunch file exists
if [ ! -f quicklaunch ]; then
  echo "quicklaunch file not found in the current directory."
  exit 1
fi

# Create a new tmux session
SESSION_NAME="bot_session"
tmux new-session -d -s $SESSION_NAME

# Read the quicklaunch file and launch each bot in a new tmux pane
FIRST=true
while IFS= read -r bot_dir; do
  if [ "$FIRST" = true ]; then
    #tmux send-keys -t $SESSION_NAME "cd $bot_dir && node index.js" C-m
    tmux send-keys -t $SESSION_NAME "cd $bot_dir" Enter
    tmux send-keys -t $SESSION_NAME "./index.js" Enter
    FIRST=false
  else
    tmux split-window -t $SESSION_NAME
    #tmux send-keys -t $SESSION_NAME "cd $bot_dir && node index.js" C-m
    tmux send-keys -t $SESSION_NAME "cd $bot_dir" Enter
    tmux send-keys -t $SESSION_NAME "./index.js" Enter
  fi
  tmux select-layout -t $SESSION_NAME tiled > /dev/null
done < quicklaunch

# Attach to the tmux session
tmux attach -t $SESSION_NAME
Comment preview
[-]JasonCarswell
1(+1|0)1 month ago

It's all Geek to me.

You may have considered this but if it could help in anyway, you could utilize whatever built in or addon functionality that HumHub has. I haven't delved in yet, but know HumHub users have lots of options like sharing files and making html pages. Maybe there's more that could also be useful. I guess it depends on your goals.

parentlinkreply
[-]x0x7
1(+1|0)1 month ago

And I realized my preload.js file made it so I have non-migratable code. I did a thing to my system that is considered very bad in the NodeJS world. I have a file that adds a lot of functions to the global variable and more functions to Arrays and to the Math library.

fsp is equal to require('fs').promises.

parentlinkreply
[-]HighQulaityDickPics0(+0|0)1 month ago

AND you just implemented the save feature. PERFECT time!

parentlinkreply