Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.048912424s FAIL (2025-07-26 23:35:00 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== equestItem_SegmentBegin Jul 26 23:34:00.737700749 (1h42m16.518634134s) [seg_pos={0 0}, upload=1568]: Begin upload segment... Jul 26 23:34:00.737772247 (1h42m16.518705616s) [seg_pos={0 0}, upload=1568]: Done begin upload segment. Jul 26 23:34:00.737799270 (1h42m16.518732657s) [noise=true , address=104.36.82.47:4523 , piece_id=CTYLPPTWOPDNEBFNBT62ADKLLOB4A7DUKARJEVDGTLJ3WTQJMCIQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.737819822 (1h42m16.518753317s) [noise=true , address=104.36.82.54:4166 , piece_id=PDKUUD2O4C6G4NOCZGKI5CDD6GFWCHJISJSYEA7CYI22DIZEGSUQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.738347939 (1h42m16.519281432s) [noise=true , address=45.91.116.9:4270 , piece_id=GS52D7IGNNUSLOMMMW7VWRRXQBWF6IHHD5JRIZZ4TL35RCOUTT2A, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.738429916 (1h42m16.519363288s) [noise=true , address=79.127.201.214:4289 , piece_id=ILR652WR4DVVPDBQWK3GSMZPAQQSLXXXBTNEW3MHJZW5CRTZ46NQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.738833033 (1h42m16.519766418s) [noise=true , address=77.90.31.17:4782 , piece_id=AGSEE4JIZCCOCCLJLADMRX3TPVG4ZFCKCNF6QEZLE5G2JO36OT7Q, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.740352024 (1h42m16.521285392s) [noise=true , address=79.127.220.101:4041 , piece_id=HPTWDVEQTZDQG6OGPTR54LQ56VAHJGRU3LGPUQHDJJ3U4HFB35FA, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.740773144 (1h42m16.521706533s) [noise=true , address=156.146.43.225:4434 , piece_id=5VMHHR3IMZBDUJAGOVTHZXKKN6J56P76QMTLRTV3XVYY2BYE5QIQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.740867890 (1h42m16.521801272s) [noise=true , address=79.127.219.38:4001 , piece_id=4VMGEJ4HSUNWOMS6GVGR7MRET42PW75BRQRFYJWNB6E5ZCMUYMGQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.741303389 (1h42m16.522236771s) [noise=true , address=79.127.220.98:4422 , piece_id=66OIKBF2AM7JBWSFSMFRYUU7BGMAMBQQ7KIWCN4AUOU4MVVAUQXA, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.741787134 (1h42m16.522720516s) [noise=true , address=104.36.82.46:4501 , piece_id=3FSD3WE32RQG6OAXVSL2YV2CH2NGUVMCWKKR4JU3JZNTOKCSZDUQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.742286052 (1h42m16.523219439s) [noise=true , address=104.36.82.56:4263 , piece_id=X6HWOLT7LHT2ME2LBS7GPX7WPBVWPZTKOJQP7E54TJ2HWHDZPCJQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.742721148 (1h42m16.52365453s) [noise=true , address=79.127.205.245:4245 , piece_id=KH3CAFF2WWLJDX3QZEK7XKOF4CRHU6LS73KJZVJNEUQPZG2YWVXA, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.743436565 (1h42m16.524369952s) [noise=true , address=79.127.205.246:4282 , piece_id=6NQRVHEYIKJDWKL6J3KU5X2Y6IPFBS6ARSM6RY4X672PXOHMZBYQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.744077419 (1h42m16.525010833s) [noise=true , address=79.127.201.215:4354 , piece_id=U5BRN732WWNRCXELQA25AGCO4VPKYSTMNZUJMUPS36YMJ6IISVFQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.744526597 (1h42m16.525459988s) [noise=true , address=79.127.205.234:4607 , piece_id=BTFFNZBUQ5AQRL35YXMW3P53D6AAYRB27DWXJ5ELPKPYK7QX2PGQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.745079232 (1h42m16.526012708s) [noise=true , address=104.36.82.55:4207 , piece_id=QV35AAUSONDCSLM2P64N7D5ZHPQHPTXAKHPTQCGTVAOXT7HG2V4Q, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.745284785 (1h42m16.52621817s) [noise=true , address=104.36.82.52:4106 , piece_id=3WNFALTDWUD2VELI3XJKOHZ44OSZZPBHIWWV3DLYHJ77UQCU3XIQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.745649491 (1h42m16.526582871s) [noise=true , address=64.49.13.138:4482 , piece_id=L7LZREKAH3SSNUZR7JXA63OVVLMQ3HW72QVIQYAPO3LQYFO6ZRPQ, seg_pos={0 0}, upload=1568]: Uploading piece... Jul 26 23:34:00.850787830 (1h42m16.631721196s) [upload=1568]: Next returned nil segment Jul 26 23:34:00.850811119 (1h42m16.631744483s) [upload=1568]: Waiting for error group managing segments... Jul 26 23:34:00.856897735 (1h42m16.637831112s) [noise=true , address=79.127.219.38:4001 , piece_id=4VMGEJ4HSUNWOMS6GVGR7MRET42PW75BRQRFYJWNB6E5ZCMUYMGQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:01.016511260 (1h42m16.797444629s) [noise=true , address=79.127.201.214:4289 , piece_id=ILR652WR4DVVPDBQWK3GSMZPAQQSLXXXBTNEW3MHJZW5CRTZ46NQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:01.333563976 (1h42m17.114497372s) [noise=true , address=79.127.205.234:4607 , piece_id=BTFFNZBUQ5AQRL35YXMW3P53D6AAYRB27DWXJ5ELPKPYK7QX2PGQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:01.347878694 (1h42m17.128812063s) [noise=true , address=79.127.205.245:4245 , piece_id=KH3CAFF2WWLJDX3QZEK7XKOF4CRHU6LS73KJZVJNEUQPZG2YWVXA, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:01.412325845 (1h42m17.193259227s) [noise=true , address=79.127.205.246:4282 , piece_id=6NQRVHEYIKJDWKL6J3KU5X2Y6IPFBS6ARSM6RY4X672PXOHMZBYQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:01.432330393 (1h42m17.213263775s) [noise=true , address=64.49.13.138:4482 , piece_id=L7LZREKAH3SSNUZR7JXA63OVVLMQ3HW72QVIQYAPO3LQYFO6ZRPQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:02.342277081 (1h42m18.123210452s) [noise=true , address=156.146.43.225:4434 , piece_id=5VMHHR3IMZBDUJAGOVTHZXKKN6J56P76QMTLRTV3XVYY2BYE5QIQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:03.623276829 (1h42m19.4042102s) [noise=true , address=79.127.220.101:4041 , piece_id=HPTWDVEQTZDQG6OGPTR54LQ56VAHJGRU3LGPUQHDJJ3U4HFB35FA, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:04.134317249 (1h42m19.915250632s) [noise=true , address=79.127.220.98:4422 , piece_id=66OIKBF2AM7JBWSFSMFRYUU7BGMAMBQQ7KIWCN4AUOU4MVVAUQXA, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:06.500220285 (1h42m22.281153655s) [noise=true , address=45.91.116.9:4270 , piece_id=GS52D7IGNNUSLOMMMW7VWRRXQBWF6IHHD5JRIZZ4TL35RCOUTT2A, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:34:43.282849015 (1h42m59.063782391s) [noise=true , address=79.127.201.215:4354 , piece_id=U5BRN732WWNRCXELQA25AGCO4VPKYSTMNZUJMUPS36YMJ6IISVFQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: <nil> Jul 26 23:35:00.662109754 (1h43m16.443043122s) [noise=true , address=104.36.82.46:4501 , piece_id=3FSD3WE32RQG6OAXVSL2YV2CH2NGUVMCWKKR4JU3JZNTOKCSZDUQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: ecclient: upload failed (node:1hmsSAiSp1CBFqKhVE2S3sFfqYYkGVAiV1GfrT4Rd7hBaFNPpT, address:104.36.82.46:4501): protocol: rpc: dial tcp 104.36.82.46:4501: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:00.662934418 (1h43m16.443867799s) [noise=true , address=77.90.31.17:4782 , piece_id=AGSEE4JIZCCOCCLJLADMRX3TPVG4ZFCKCNF6QEZLE5G2JO36OT7Q, seg_pos={0 0}, upload=1568]: Done uploading piece. err: ecclient: upload failed (node:12FtB4bKTHntdUQxF8G5o9nT73oS2xB7fgD44BkRfKDNFyZnKCk, address:77.90.31.17:4782): protocol: rpc: dial tcp 77.90.31.17:4782: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:00.663110677 (1h43m16.444044057s) [noise=true , address=104.36.82.56:4263 , piece_id=X6HWOLT7LHT2ME2LBS7GPX7WPBVWPZTKOJQP7E54TJ2HWHDZPCJQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: ecclient: upload failed (node:12tLW4r8AQKLCbSWZoNUVi5v2YtF8NthUmX81TLQn8G8TotuByG, address:104.36.82.56:4263): protocol: rpc: dial tcp 104.36.82.56:4263: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:00.663225896 (1h43m16.444159278s) [noise=true , address=104.36.82.55:4207 , piece_id=QV35AAUSONDCSLM2P64N7D5ZHPQHPTXAKHPTQCGTVAOXT7HG2V4Q, seg_pos={0 0}, upload=1568]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: rpc: dial tcp 104.36.82.55:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:00.663349243 (1h43m16.444282624s) [noise=true , address=104.36.82.54:4166 , piece_id=PDKUUD2O4C6G4NOCZGKI5CDD6GFWCHJISJSYEA7CYI22DIZEGSUQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: ecclient: upload failed (node:1BV7mWGAwKMXzmZBc3qAQFUXYyWtgQC7QEfG54YJgyJ4U3tFiv, address:104.36.82.54:4166): protocol: rpc: dial tcp 104.36.82.54:4166: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:00.663492963 (1h43m16.444426345s) [noise=true , address=104.36.82.47:4523 , piece_id=CTYLPPTWOPDNEBFNBT62ADKLLOB4A7DUKARJEVDGTLJ3WTQJMCIQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: ecclient: upload failed (node:12pADNfAT7ccugc1jKnTsk2xiw1aSEpuoKZSRosexzHowDmDCWV, address:104.36.82.47:4523): protocol: write tcp 159.223.143.39:36124->104.36.82.47:4523: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:00.663529961 (1h43m16.444463339s) [noise=true , address=104.36.82.52:4106 , piece_id=3WNFALTDWUD2VELI3XJKOHZ44OSZZPBHIWWV3DLYHJ77UQCU3XIQ, seg_pos={0 0}, upload=1568]: Done uploading piece. err: ecclient: upload failed (node:12E2hMuAUbJyh5F4yey1EtRLYhzr29wxs47qY3zGxmGvFNUctk2, address:104.36.82.52:4106): protocol: write tcp 159.223.143.39:39964->104.36.82.52:4106: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:00.663585416 (1h43m16.444518824s) [seg_pos={0 0}, upload=1568]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:35:00.706519445 (1h43m16.487452811s) [upload=1568]: Done uploading segments... Jul 26 23:35:00.706551058 (1h43m16.487484425s) [upload=1568]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.048954187s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 35 out of 60 is failed (threshold=3) [...................X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.057867129s FAIL (2025-07-26 23:34:56 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.057894643s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [..X.X..........XX...X...X..XX.X...X..X.......X...X....XX....] duration: 14.888515279s FAIL (2025-07-26 23:34:50 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 14.888544156s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 33 out of 60 is failed (threshold=3) [...........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.028445732s FAIL (2025-07-26 23:34:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== Covw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367198603 (1h41m56.148131979s) [noise=true , address=104.36.82.5:4839 , piece_id=USX2JFKUV34UZUV4VU6KUBGPUK7K6FITP574RNCEXBEWKKVADKJA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:48926->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367235055 (1h41m56.148168436s) [seg_pos={0 0}, upload=1557]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:33:40.418644019 (1h41m56.199577399s) [upload=1557]: Done uploading segments... Jul 26 23:33:40.418702283 (1h41m56.199635662s) [upload=1557]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 26 23:34:10.117611634 (1h42m25.898544996s) [upload=1569]: Starting upload Jul 26 23:34:10.117710905 (1h42m25.898644283s) [upload=1569]: Done starting upload Jul 26 23:34:10.117866573 (1h42m25.898799943s) [upload=1569]: Uploading segments... Jul 26 23:34:10.118336221 (1h42m25.899269633s) [upload=1569]: Got next segment. Inline: false Jul 26 23:34:10.118407651 (1h42m25.899341028s) [upload=1569]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:34:10.118415198 (1h42m25.899348586s) [upload=1569]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:34:10.151847318 (1h42m25.932780734s) [seg_pos={0 0}, upload=1569]: Begin upload segment... Jul 26 23:34:10.151925735 (1h42m25.932859114s) [seg_pos={0 0}, upload=1569]: Done begin upload segment. Jul 26 23:34:10.151963567 (1h42m25.932896959s) [noise=true , address=45.91.116.5:4298 , piece_id=GIAYF6FZRWFG7SUCCBPFDAJFNSH3IIRHL3IRVYOD33VWJ3YOTTWQ, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.151980790 (1h42m25.932914169s) [noise=true , address=45.91.116.5:4297 , piece_id=NDI2Z6KM7WDCSN2XR52M6LCF53ECEI5MH2D7OUOKADOMVFK2FMRQ, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.152588724 (1h42m25.933522108s) [noise=true , address=45.91.116.9:4261 , piece_id=YRS5P675XSJUPIDMPBEABQ6TPKA2LKXEHQJ2POAZI2GWTPCUPRYQ, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.152729338 (1h42m25.933662726s) [noise=true , address=77.90.31.19:4842 , piece_id=XXQA2KLI7OTTNZJVXPCAB4QIJA4FH5KPEN7O77OYQIQIFNODXU5A, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.152939491 (1h42m25.933872874s) [noise=true , address=45.91.116.6:4974 , piece_id=WXG6DG6WQ2S74JYOPBJEXXRCQSTISMMYGKACXIH72IYI5EW6JLXQ, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.153200339 (1h42m25.934133877s) [noise=true , address=45.91.116.12:4227 , piece_id=HY54R26JDN2XVDXXJKFUTJTK2MUFPG73XOIJFOZ76MXSLZJFSMQA, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.153330981 (1h42m25.934264362s) [noise=true , address=77.90.31.15:4697 , piece_id=2EQV5BWVN6QFAMMSMPUMUWOTV4TF4AI37LRVB255XOEYYCZTTMAA, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.153670583 (1h42m25.934603973s) [noise=true , address=104.36.82.10:4078 , piece_id=SNMFIG3O3ETCRHCPOZVNSVTRCOVJBJUYI63M7DCFA5R2BNTVQRJQ, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.153691435 (1h42m25.934624799s) [noise=true , address=45.91.116.7:4010 , piece_id=AIM76ZBZCXJZDTHX7MN3V5K3PP4MZGR2ZCO236G2NN75W6XIH53A, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.154154309 (1h42m25.935087696s) [noise=true , address=104.36.82.2:4730 , piece_id=GVJQ5BKZJNDNUS6YFGTSKDTLRASCKRVNE6NTQGTKPB5FIVAJMQPA, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.154642256 (1h42m25.935575643s) [noise=true , address=45.91.116.12:4225 , piece_id=QQNIUZYBMGM4DIIVEWNXNNOPRSYOUAD5JUDZ5ZDBGYZD3VKVRATA, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.155137383 (1h42m25.936070762s) [noise=true , address=45.91.116.6:4975 , piece_id=46VNMTMFVXMKHY2X52WGM2TMF6SOWHHDVYE2PDH34G5477XJVRDQ, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.155551019 (1h42m25.936484396s) [noise=true , address=45.91.116.13:4153 , piece_id=V3B6S53NAQI2U7U2S4VGUBOXDQTUCVSD25EMV4RF27A4OVK7OF3A, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.156101107 (1h42m25.937034489s) [noise=true , address=104.36.82.12:4150 , piece_id=X2K72WBWRBLTW4EE2ILEAMPTRQIHMRWXFMWMK2L6YL5HOTBHCZ5Q, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.156611794 (1h42m25.93754518s) [noise=true , address=45.91.116.4:4939 , piece_id=54UNTXSCAEYVF2ZP4GJY44L2UL4SPBQLXEEJ7V6RJH2YYXN7K7BQ, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.157022833 (1h42m25.93795622s) [noise=true , address=104.36.82.16:4294 , piece_id=Z7AC73U2VQPXOQC75WIMGVVFVYWXGZCIYTYPJZWUOHD2MAN4RBGA, seg_pos={0 0}, upload=1569]: Uploading piece... Jul 26 23:34:10.278956870 (1h42m26.05989024s) [upload=1569]: Next returned nil segment Jul 26 23:34:10.278976555 (1h42m26.059909937s) [upload=1569]: Waiting for error group managing segments... Jul 26 23:34:10.283273193 (1h42m26.064206589s) [noise=true , address=45.91.116.4:4939 , piece_id=54UNTXSCAEYVF2ZP4GJY44L2UL4SPBQLXEEJ7V6RJH2YYXN7K7BQ, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.283736322 (1h42m26.064669691s) [noise=true , address=45.91.116.6:4974 , piece_id=WXG6DG6WQ2S74JYOPBJEXXRCQSTISMMYGKACXIH72IYI5EW6JLXQ, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.284133013 (1h42m26.065066382s) [noise=true , address=45.91.116.12:4227 , piece_id=HY54R26JDN2XVDXXJKFUTJTK2MUFPG73XOIJFOZ76MXSLZJFSMQA, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.284529969 (1h42m26.065463332s) [noise=true , address=77.90.31.15:4697 , piece_id=2EQV5BWVN6QFAMMSMPUMUWOTV4TF4AI37LRVB255XOEYYCZTTMAA, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.284962625 (1h42m26.065896004s) [noise=true , address=45.91.116.13:4153 , piece_id=V3B6S53NAQI2U7U2S4VGUBOXDQTUCVSD25EMV4RF27A4OVK7OF3A, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.284990021 (1h42m26.065923393s) [noise=true , address=45.91.116.7:4010 , piece_id=AIM76ZBZCXJZDTHX7MN3V5K3PP4MZGR2ZCO236G2NN75W6XIH53A, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.286004298 (1h42m26.066937667s) [noise=true , address=45.91.116.6:4975 , piece_id=46VNMTMFVXMKHY2X52WGM2TMF6SOWHHDVYE2PDH34G5477XJVRDQ, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.301316119 (1h42m26.082249509s) [noise=true , address=45.91.116.12:4225 , piece_id=QQNIUZYBMGM4DIIVEWNXNNOPRSYOUAD5JUDZ5ZDBGYZD3VKVRATA, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.550182785 (1h42m26.331116155s) [noise=true , address=45.91.116.9:4261 , piece_id=YRS5P675XSJUPIDMPBEABQ6TPKA2LKXEHQJ2POAZI2GWTPCUPRYQ, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.719621993 (1h42m26.50055536s) [noise=true , address=45.91.116.5:4298 , piece_id=GIAYF6FZRWFG7SUCCBPFDAJFNSH3IIRHL3IRVYOD33VWJ3YOTTWQ, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:10.733586459 (1h42m26.51451983s) [noise=true , address=45.91.116.5:4297 , piece_id=NDI2Z6KM7WDCSN2XR52M6LCF53ECEI5MH2D7OUOKADOMVFK2FMRQ, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:12.318890457 (1h42m28.099823825s) [noise=true , address=77.90.31.19:4842 , piece_id=XXQA2KLI7OTTNZJVXPCAB4QIJA4FH5KPEN7O77OYQIQIFNODXU5A, seg_pos={0 0}, upload=1569]: Done uploading piece. err: <nil> Jul 26 23:34:40.057141063 (1h42m55.83807444s) [noise=true , address=104.36.82.10:4078 , piece_id=SNMFIG3O3ETCRHCPOZVNSVTRCOVJBJUYI63M7DCFA5R2BNTVQRJQ, seg_pos={0 0}, upload=1569]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:40.057236375 (1h42m55.838169741s) [noise=true , address=104.36.82.2:4730 , piece_id=GVJQ5BKZJNDNUS6YFGTSKDTLRASCKRVNE6NTQGTKPB5FIVAJMQPA, seg_pos={0 0}, upload=1569]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:40.057327362 (1h42m55.838260728s) [noise=true , address=104.36.82.16:4294 , piece_id=Z7AC73U2VQPXOQC75WIMGVVFVYWXGZCIYTYPJZWUOHD2MAN4RBGA, seg_pos={0 0}, upload=1569]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: rpc: dial tcp 104.36.82.16:4294: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:40.057396771 (1h42m55.838330137s) [noise=true , address=104.36.82.12:4150 , piece_id=X2K72WBWRBLTW4EE2ILEAMPTRQIHMRWXFMWMK2L6YL5HOTBHCZ5Q, seg_pos={0 0}, upload=1569]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:40600->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:40.057416370 (1h42m55.838349747s) [seg_pos={0 0}, upload=1569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:34:40.082099388 (1h42m55.863032755s) [upload=1569]: Done uploading segments... Jul 26 23:34:40.082136123 (1h42m55.863069512s) [upload=1569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.028571482s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [...X.X..........XX...X...X..XX.X...X..X.......X...X....XX...] duration: 43.630721457s FAIL (2025-07-26 23:34:28 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.63074948s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [........................XXX.X....XX.X.X.X.......XXX...XX....] duration: 2.014707975s FAIL (2025-07-26 23:34:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.014764827s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.031070205s FAIL (2025-07-26 23:34:00 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== WQFDADOX3EAQ, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.641611597 (1h41m16.42254498s) [noise=true , address=77.90.31.19:4871 , piece_id=JBUXIBPSEADTTRVUJ3E4HN7UCFGE5Y4SZ4CTU5AVZW7AABTM6IFQ, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.641967837 (1h41m16.422901231s) [noise=true , address=104.36.82.47:4530 , piece_id=WKQRVPUDU3GWIRDHHRSELERDHCLX6FGDUGTEIIEAO33LSZSJOCOQ, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.642685688 (1h41m16.423619082s) [noise=true , address=77.90.31.18:4821 , piece_id=NOR4OYLYORUBVORQX6T7HQEEIWFANUP7FZM736VPMOIQWU7IRDYQ, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.642890793 (1h41m16.423824177s) [noise=true , address=104.36.82.54:4177 , piece_id=62HU5KO42VQL7B25ZLIEOXW5FV5ALO6Y5UTXCSJVXS6MTFSDTIMQ, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.643487814 (1h41m16.424421194s) [noise=true , address=79.127.201.215:4355 , piece_id=SXPU2BGAM7MYIWR45FPIHI4WCKHTQKBEKDOLALUPMLJGHZ6IU23Q, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.643536526 (1h41m16.424469912s) [noise=true , address=77.90.31.17:4787 , piece_id=EP7BFHHIFK2V7FAEK2EIN5SZLCHDDVAKT37U2AHUHHIXOTVNEY4A, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.643940021 (1h41m16.424873411s) [noise=true , address=79.127.205.234:4607 , piece_id=PZ7OYTUK5IHIYTVDSTXQYRZEWOKHG3SNUJNY75P4G3VCGPFMI6BQ, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.643966746 (1h41m16.424900129s) [noise=true , address=79.127.220.98:4422 , piece_id=WAVGYZ43ZKCDLXP7OVTRH6SIFPMOGGQE2DB4KJEGU3F7IBDBWBOQ, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.644465625 (1h41m16.425399087s) [noise=true , address=104.36.82.46:4513 , piece_id=X6JAJ7LP52L2ZU6H6POXT2SIEOKV4R27FUQTJOTO24TO2CJMKHNA, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.644498294 (1h41m16.425431679s) [noise=true , address=104.36.82.22:4514 , piece_id=2YDELYDN44OES2KUVXHVBQ3ZRLOJ4G7PAKGRBVBDY4UXMFRHJPLQ, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.644942162 (1h41m16.425875551s) [noise=true , address=64.49.13.137:4502 , piece_id=GDV67RIJCJV6XKGOPTHLZLXZRLYRB7X57WM7LYM6AKXDC4QB4F6A, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.645340487 (1h41m16.426273876s) [noise=true , address=77.90.31.12:4602 , piece_id=UETVWAEM64VE6ELYYKCDNKPIBAXRJNOTK2AS6DZI26P53FL3YXSA, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.645726110 (1h41m16.4266595s) [noise=true , address=104.36.82.55:4202 , piece_id=HCPA3WWF7POYGJCYZC5Z7QBRELSOG7G4LVPD6PSJX4BNTRCK7B3Q, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.647326218 (1h41m16.428259607s) [noise=true , address=79.127.219.38:4005 , piece_id=2KIIIHAK2F4FSIHHFKNSJ2OX4WG777PKRE3YCNXMBDHHPHNOE7AA, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.647907593 (1h41m16.428840955s) [noise=true , address=77.90.31.21:4906 , piece_id=YISCG44QDDGPOBUGQTMBXT2VFYADKCZJ2N2JRUZGZVNBIJLUJ3VQ, seg_pos={0 0}, upload=1554]: Uploading piece... Jul 26 23:33:00.750777473 (1h41m16.531710852s) [upload=1554]: Next returned nil segment Jul 26 23:33:00.750797755 (1h41m16.531731123s) [upload=1554]: Waiting for error group managing segments... Jul 26 23:33:00.762018232 (1h41m16.542951597s) [noise=true , address=79.127.219.38:4005 , piece_id=2KIIIHAK2F4FSIHHFKNSJ2OX4WG777PKRE3YCNXMBDHHPHNOE7AA, seg_pos={0 0}, upload=1554]: Done uploading piece. err: <nil> Jul 26 23:33:00.775914020 (1h41m16.556847399s) [noise=true , address=77.90.31.21:4906 , piece_id=YISCG44QDDGPOBUGQTMBXT2VFYADKCZJ2N2JRUZGZVNBIJLUJ3VQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: <nil> Jul 26 23:33:00.837314740 (1h41m16.618248105s) [noise=true , address=156.146.43.225:4434 , piece_id=TDWDSLZMIKDYDDH43NQNL3N4QZCUOSOETEGIQV766XFPBLFG74PQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:12WJ1b6sKJG4NcqmNnvSLVw1e8Us6phnN5UpJY2tvo38pdmhFLp, address:156.146.43.225:4434): protocol: storage node overloaded, request limit: 2000 Jul 26 23:33:01.233662901 (1h41m17.014596278s) [noise=true , address=79.127.205.234:4607 , piece_id=PZ7OYTUK5IHIYTVDSTXQYRZEWOKHG3SNUJNY75P4G3VCGPFMI6BQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: <nil> Jul 26 23:33:01.986856055 (1h41m17.767789441s) [noise=true , address=64.49.13.137:4502 , piece_id=GDV67RIJCJV6XKGOPTHLZLXZRLYRB7X57WM7LYM6AKXDC4QB4F6A, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:12YXATeBYy7Q21fYraVJB5aw9zwEXh86FfdHGHFXANEHGGCPyh6, address:64.49.13.137:4502): protocol: storage node overloaded, request limit: 2000 Jul 26 23:33:03.615690829 (1h41m19.396624196s) [noise=true , address=79.127.220.98:4422 , piece_id=WAVGYZ43ZKCDLXP7OVTRH6SIFPMOGGQE2DB4KJEGU3F7IBDBWBOQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: <nil> Jul 26 23:33:07.962746934 (1h41m23.743680302s) [noise=true , address=77.90.31.19:4871 , piece_id=JBUXIBPSEADTTRVUJ3E4HN7UCFGE5Y4SZ4CTU5AVZW7AABTM6IFQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: <nil> Jul 26 23:33:13.197228992 (1h41m28.978162361s) [noise=true , address=79.127.201.215:4355 , piece_id=SXPU2BGAM7MYIWR45FPIHI4WCKHTQKBEKDOLALUPMLJGHZ6IU23Q, seg_pos={0 0}, upload=1554]: Done uploading piece. err: <nil> Jul 26 23:34:00.567666622 (1h42m16.348600005s) [noise=true , address=104.36.82.54:4177 , piece_id=62HU5KO42VQL7B25ZLIEOXW5FV5ALO6Y5UTXCSJVXS6MTFSDTIMQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: rpc: dial tcp 104.36.82.54:4177: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:00.567774109 (1h42m16.348707477s) [noise=true , address=104.36.82.55:4202 , piece_id=HCPA3WWF7POYGJCYZC5Z7QBRELSOG7G4LVPD6PSJX4BNTRCK7B3Q, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:104.36.82.55:4202): protocol: rpc: dial tcp 104.36.82.55:4202: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:00.567858950 (1h42m16.348792332s) [noise=true , address=104.36.82.47:4530 , piece_id=WKQRVPUDU3GWIRDHHRSELERDHCLX6FGDUGTEIIEAO33LSZSJOCOQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:1WVxMFx9jm6CT7GcXg2qgq3Be9UPFf1KZKssqufSk2ebdBBmxR, address:104.36.82.47:4530): protocol: rpc: dial tcp 104.36.82.47:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:00.567927527 (1h42m16.348860906s) [noise=true , address=77.90.31.17:4787 , piece_id=EP7BFHHIFK2V7FAEK2EIN5SZLCHDDVAKT37U2AHUHHIXOTVNEY4A, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:1Lietonb5aVbaZQt6wQ2EoxwJkSEn9oL9Dwyrc9mTyomEKikNt, address:77.90.31.17:4787): protocol: rpc: dial tcp 77.90.31.17:4787: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:00.568002098 (1h42m16.348935461s) [noise=true , address=104.36.82.22:4514 , piece_id=2YDELYDN44OES2KUVXHVBQ3ZRLOJ4G7PAKGRBVBDY4UXMFRHJPLQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:12Nh9nHAnKWgu1ggCAZ7WcUadbvsPqAgbqzGrBnSaDMqDhquQ4R, address:104.36.82.22:4514): protocol: rpc: dial tcp 104.36.82.22:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:00.568160727 (1h42m16.34909409s) [noise=true , address=77.90.31.12:4602 , piece_id=UETVWAEM64VE6ELYYKCDNKPIBAXRJNOTK2AS6DZI26P53FL3YXSA, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:1GtnQbBbyjZQHrAXhG5r4Kt4MYyiUtEgRnj8VsSjA2q4bGkyjf, address:77.90.31.12:4602): protocol: rpc: dial tcp 77.90.31.12:4602: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:00.568233155 (1h42m16.349166523s) [noise=true , address=104.36.82.53:4152 , piece_id=GYMX5GSFR7SATHB7PLZM2PTR5F5RUOJSS3Y4YREWWQFDADOX3EAQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:127p3Fibbp6XQSewgrTiBAn45TSviFfjSGpv92tGLxsDSKWPuUe, address:104.36.82.53:4152): protocol: write tcp 159.223.143.39:60046->104.36.82.53:4152: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:00.568407586 (1h42m16.34934095s) [noise=true , address=104.36.82.46:4513 , piece_id=X6JAJ7LP52L2ZU6H6POXT2SIEOKV4R27FUQTJOTO24TO2CJMKHNA, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:34:00.568556099 (1h42m16.349489463s) [noise=true , address=77.90.31.18:4821 , piece_id=NOR4OYLYORUBVORQX6T7HQEEIWFANUP7FZM736VPMOIQWU7IRDYQ, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:12dSS7NcpaDB98oZdBM6vFe56mejA8fJZKA5YvERXTD9qqXVPdH, address:77.90.31.18:4821): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:34:00.568752939 (1h42m16.34968632s) [noise=true , address=104.36.82.56:4246 , piece_id=RLKKQRD4Q4TQZ5DJCTWEB4KOD2IZYB6ONKACYIWN76VD5JVI533Q, seg_pos={0 0}, upload=1554]: Done uploading piece. err: ecclient: upload failed (node:1XtKegw9zcPDGZu9z7STKgAzwx5KFrkB9qCgdNUe6XqZ6tzzj9, address:104.36.82.56:4246): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:34:00.568775348 (1h42m16.34970873s) [seg_pos={0 0}, upload=1554]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 26 23:34:00.595410818 (1h42m16.376344187s) [upload=1554]: Done uploading segments... Jul 26 23:34:00.595443660 (1h42m16.376377032s) [upload=1554]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.031090347s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 34 out of 60 is failed (threshold=3) [....................X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.026668982s FAIL (2025-07-26 23:33:56 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.026721375s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [....X.X..........XX...X...X..XX.X...X..X.......X...X....XX..] duration: 45.511306563s FAIL (2025-07-26 23:33:44 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 45.511338436s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.........................XXX.X....XX.X.X.X.......XXX...XX...] duration: 637.747882ms FAIL (2025-07-26 23:33:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 637.778114ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 32 out of 60 is failed (threshold=3) [............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.054525039s FAIL (2025-07-26 23:33:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== ]: Done starting upload Jul 26 23:33:10.399859122 (1h41m26.180792491s) [upload=1557]: Uploading segments... Jul 26 23:33:10.399921012 (1h41m26.1808544s) [upload=1557]: Got next segment. Inline: false Jul 26 23:33:10.399963477 (1h41m26.180896863s) [upload=1557]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:33:10.399970489 (1h41m26.180903875s) [upload=1557]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:33:10.427358387 (1h41m26.208291775s) [seg_pos={0 0}, upload=1557]: Begin upload segment... Jul 26 23:33:10.427454451 (1h41m26.208387813s) [seg_pos={0 0}, upload=1557]: Done begin upload segment. Jul 26 23:33:10.427500329 (1h41m26.208433713s) [noise=true , address=45.91.116.6:4975 , piece_id=DMZSXLD7QRBSM5O46R23HMOXNTHVJN7I3RGR44GYRUIZ3SHHDUZA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.427521509 (1h41m26.208454894s) [noise=true , address=104.36.82.16:4293 , piece_id=4O5IUHFBXARTXNMYUQT4YAS2II2R7YHCCZ5N6VDZJQ2IMOWGVAKQ, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.428258294 (1h41m26.209191683s) [noise=true , address=104.36.82.16:4295 , piece_id=ORU44S4XE4UPECW6ZHMTDVAQEWKV7WCK4YUALWR2NWCKVP6OXAPA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.428278023 (1h41m26.209211384s) [noise=true , address=104.36.82.9:4005 , piece_id=4GY4REWU4USJ7CRTV4GVDXXLB52ELWT67FTY4CLBZTMQJSVBE4TQ, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.428671971 (1h41m26.209605349s) [noise=true , address=45.91.116.9:4261 , piece_id=R47BU7WN26UUP6PA6DKFOS2RVY2HZX6W5OG2AMR22RJEQXQIQTPA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.428753539 (1h41m26.209686908s) [noise=true , address=45.91.116.13:4153 , piece_id=B7HORTLQWCV7QJBNIUT3G6CNNFPCX3DW3LEUGESGJRWAVW77X2OA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.429075533 (1h41m26.210008914s) [noise=true , address=45.91.116.5:4297 , piece_id=UZ5ARATYV422WL2QODOIJ2HCOZ7Q4BVWCY7VIW2KJBA3A443CPZA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.429561053 (1h41m26.210494431s) [noise=true , address=104.36.82.13:4187 , piece_id=J2FC4MOJOEAWU6GRSBOTGJ7JPXRZLXF37YEJVWSYESWPST3VXWNQ, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.430018688 (1h41m26.210952066s) [noise=true , address=104.36.82.5:4839 , piece_id=USX2JFKUV34UZUV4VU6KUBGPUK7K6FITP574RNCEXBEWKKVADKJA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.430069970 (1h41m26.21100334s) [noise=true , address=104.36.82.12:4150 , piece_id=J3MGMI5FC26BHG2GQE2YAVZEH3ZULCF2IYNWX6CWBQGJ3WPPDOXA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.430442592 (1h41m26.21137598s) [noise=true , address=104.36.82.8:4969 , piece_id=G5TLXWEUIKPG2V7PTVYWPMRVTULCW3O7I74VUKIMVWNSIXMBUP6Q, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.430815607 (1h41m26.211749001s) [noise=true , address=104.36.82.7:4912 , piece_id=NNJN2XBUJ2SZLGKPN76RJCWZVFGFJ4KDR3BVO3QZXZ3S3KJDPWGA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.431084578 (1h41m26.212017962s) [noise=true , address=45.91.116.14:4189 , piece_id=FQHSP72JSQMQZ5UVPVIJ4QNBCWFLD2KQJYNXPLKCCQRFBEBG2PBA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.431634704 (1h41m26.212568086s) [noise=true , address=45.91.116.14:4191 , piece_id=WVPCSQLMPY7F4DILLY56OFUFALZEX34R3RLIDJHMER26FLP4J5NA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.432565394 (1h41m26.21349877s) [noise=true , address=45.91.116.12:4227 , piece_id=FG2ZVMM72EZJH4SHSIYGYT5TPCQNYJNDBKAMGVFMII5AW7DZV2IA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.433293254 (1h41m26.21422662s) [noise=true , address=104.36.82.12:4149 , piece_id=HFGQU3APX4IO6IY2FP43OIXA5UFO4J3O5TKUAEFESV3465VVEBUA, seg_pos={0 0}, upload=1557]: Uploading piece... Jul 26 23:33:10.532771490 (1h41m26.313704867s) [upload=1557]: Next returned nil segment Jul 26 23:33:10.532791528 (1h41m26.313724915s) [upload=1557]: Waiting for error group managing segments... Jul 26 23:33:10.537134935 (1h41m26.318068298s) [noise=true , address=45.91.116.12:4227 , piece_id=FG2ZVMM72EZJH4SHSIYGYT5TPCQNYJNDBKAMGVFMII5AW7DZV2IA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: <nil> Jul 26 23:33:10.537246745 (1h41m26.318180114s) [noise=true , address=45.91.116.6:4975 , piece_id=DMZSXLD7QRBSM5O46R23HMOXNTHVJN7I3RGR44GYRUIZ3SHHDUZA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: <nil> Jul 26 23:33:10.537693491 (1h41m26.318626855s) [noise=true , address=45.91.116.13:4153 , piece_id=B7HORTLQWCV7QJBNIUT3G6CNNFPCX3DW3LEUGESGJRWAVW77X2OA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: <nil> Jul 26 23:33:10.673740835 (1h41m26.454674213s) [noise=true , address=45.91.116.5:4297 , piece_id=UZ5ARATYV422WL2QODOIJ2HCOZ7Q4BVWCY7VIW2KJBA3A443CPZA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: <nil> Jul 26 23:33:10.687782734 (1h41m26.468716115s) [noise=true , address=45.91.116.9:4261 , piece_id=R47BU7WN26UUP6PA6DKFOS2RVY2HZX6W5OG2AMR22RJEQXQIQTPA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: <nil> Jul 26 23:33:10.773600304 (1h41m26.55453367s) [noise=true , address=45.91.116.14:4191 , piece_id=WVPCSQLMPY7F4DILLY56OFUFALZEX34R3RLIDJHMER26FLP4J5NA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: <nil> Jul 26 23:33:10.780943150 (1h41m26.561876529s) [noise=true , address=45.91.116.14:4189 , piece_id=FQHSP72JSQMQZ5UVPVIJ4QNBCWFLD2KQJYNXPLKCCQRFBEBG2PBA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: <nil> Jul 26 23:33:40.366858867 (1h41m56.147792246s) [noise=true , address=104.36.82.16:4295 , piece_id=ORU44S4XE4UPECW6ZHMTDVAQEWKV7WCK4YUALWR2NWCKVP6OXAPA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.366938462 (1h41m56.147871826s) [noise=true , address=104.36.82.9:4005 , piece_id=4GY4REWU4USJ7CRTV4GVDXXLB52ELWT67FTY4CLBZTMQJSVBE4TQ, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367052012 (1h41m56.147985388s) [noise=true , address=104.36.82.12:4149 , piece_id=HFGQU3APX4IO6IY2FP43OIXA5UFO4J3O5TKUAEFESV3465VVEBUA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367068681 (1h41m56.148002047s) [noise=true , address=104.36.82.13:4187 , piece_id=J2FC4MOJOEAWU6GRSBOTGJ7JPXRZLXF37YEJVWSYESWPST3VXWNQ, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367124295 (1h41m56.148057673s) [noise=true , address=104.36.82.16:4293 , piece_id=4O5IUHFBXARTXNMYUQT4YAS2II2R7YHCCZ5N6VDZJQ2IMOWGVAKQ, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: rpc: dial tcp 104.36.82.16:4293: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367163255 (1h41m56.148096621s) [noise=true , address=104.36.82.12:4150 , piece_id=J3MGMI5FC26BHG2GQE2YAVZEH3ZULCF2IYNWX6CWBQGJ3WPPDOXA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:50168->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367174765 (1h41m56.148108133s) [noise=true , address=104.36.82.8:4969 , piece_id=G5TLXWEUIKPG2V7PTVYWPMRVTULCW3O7I74VUKIMVWNSIXMBUP6Q, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367188533 (1h41m56.148121906s) [noise=true , address=104.36.82.7:4912 , piece_id=NNJN2XBUJ2SZLGKPN76RJCWZVFGFJ4KDR3BVO3QZXZ3S3KJDPWGA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367198603 (1h41m56.148131979s) [noise=true , address=104.36.82.5:4839 , piece_id=USX2JFKUV34UZUV4VU6KUBGPUK7K6FITP574RNCEXBEWKKVADKJA, seg_pos={0 0}, upload=1557]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:48926->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:40.367235055 (1h41m56.148168436s) [seg_pos={0 0}, upload=1557]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:33:40.418644019 (1h41m56.199577399s) [upload=1557]: Done uploading segments... Jul 26 23:33:40.418702283 (1h41m56.199635662s) [upload=1557]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.054554625s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..........................XXX.X....XX.X.X.X.......XXX...XX..] duration: 1.933483184s FAIL (2025-07-26 23:33:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.934159436s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 48 out of 60 is failed (threshold=3) [.XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.030649888s FAIL (2025-07-26 23:33:00 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== upload=1537]: Uploading piece... Jul 26 23:32:00.503791147 (1h40m16.284724536s) [noise=true , address=156.146.43.225:4448 , piece_id=WCQMBCF2CUOSVQASPGY6THK2PACIDKPLLHAEHIINOOI66IOCY76Q, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.504200430 (1h40m16.285133819s) [noise=true , address=77.90.31.10:4523 , piece_id=RTT3CJBI7WVM4PMMU5RXVOFXVF5XST7MA73SJZGH2QXUA7FUFJCQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.504606966 (1h40m16.28554035s) [noise=true , address=79.127.205.234:4605 , piece_id=MJRU5HAPBF6ORRB4Z6FFWSI4KNMPIOOGRQMYGRSQOOUBYYP3AKUQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.504990011 (1h40m16.285923394s) [noise=true , address=77.90.31.18:4831 , piece_id=7AGKK7XRXGSRCDVU3YJGS64OVZ6NRQWUSOWGKGKSA4F3DPKUUKUQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.505360782 (1h40m16.286294163s) [noise=true , address=79.127.201.215:4344 , piece_id=XIKKXCYS46JFDZW7RMNC2HIVNC3YWB6THQ2TJAMNVJ4ULFCHTCKQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.505733748 (1h40m16.286667112s) [noise=true , address=104.36.82.46:4496 , piece_id=373IDZNHVH6QHAY6KZDDEFO2IDB4FFHCNFF7KXHDF6AOV366PHUQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.506097792 (1h40m16.287031192s) [noise=true , address=104.36.82.54:4167 , piece_id=PYTSSV6KF2RDQGCSAA26LFU5CM7FERKLMIHNBNAUUXJYV54F47GA, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.506503500 (1h40m16.287436889s) [noise=true , address=104.36.82.10:4084 , piece_id=YJBAKYBYGA7ZNSKTKZHSPLXDBULTRQNXZ6P5FVQIBZ6E6QUDDIQQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.506884663 (1h40m16.287818053s) [noise=true , address=104.36.82.47:4532 , piece_id=G7V3LGUEFMNTJ5KRDHH6J5Y2VSAPJ6GFSR7Q7VNOBK5GXWJXGU7Q, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.507249436 (1h40m16.28818282s) [noise=true , address=45.91.116.9:4277 , piece_id=OX6EYFNEJCSQD7RDYEF7GDHKAAUNHN237EKYHPXQ465UW4YISCOA, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.507562406 (1h40m16.288495794s) [noise=true , address=77.90.31.21:4893 , piece_id=NL3ZRIJKSMILTMNUW2RGQLHE6OOH2EARLIBAMGQUKUWNWTAVPATQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.507994281 (1h40m16.288927672s) [noise=true , address=104.36.82.53:4147 , piece_id=MHQE4C2UOOKWTOSIC3VT4DOG74BWAI225JW3SF7WSXLHZ6OR4H6A, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.508539751 (1h40m16.28947314s) [noise=true , address=77.90.31.15:4723 , piece_id=B3OPFMMS3DBNKXGGPZAXYI7VKOSDEMJFO3I2QLLIF4CNYGGM3XBQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.509112523 (1h40m16.290045906s) [noise=true , address=104.36.82.55:4199 , piece_id=UIUKCUJY3RIV3KNTAOYZPPOUCNKJKND3BBQSNHW2D2IN5BANTZ6A, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.509543680 (1h40m16.29047706s) [noise=true , address=64.49.13.140:4508 , piece_id=EIHQTWDL72BNUCHSLGILXSZRREXXXBBRIUC3L5S3SZ6I37UYSFAQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.510018545 (1h40m16.290951924s) [noise=true , address=104.36.82.56:4230 , piece_id=5LBVPKYLT652M2K4FIDNJGIHU3R7Z4FW3UPGITE5RZCNQMOHGZJQ, seg_pos={0 0}, upload=1537]: Uploading piece... Jul 26 23:32:00.677960694 (1h40m16.458894073s) [upload=1537]: Next returned nil segment Jul 26 23:32:00.677980641 (1h40m16.458914021s) [upload=1537]: Waiting for error group managing segments... Jul 26 23:32:00.724065189 (1h40m16.504998561s) [noise=true , address=77.90.31.10:4523 , piece_id=RTT3CJBI7WVM4PMMU5RXVOFXVF5XST7MA73SJZGH2QXUA7FUFJCQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: <nil> Jul 26 23:32:00.724663736 (1h40m16.505597113s) [noise=true , address=77.90.31.21:4893 , piece_id=NL3ZRIJKSMILTMNUW2RGQLHE6OOH2EARLIBAMGQUKUWNWTAVPATQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: <nil> Jul 26 23:32:00.724957173 (1h40m16.505890538s) [noise=true , address=77.90.31.15:4723 , piece_id=B3OPFMMS3DBNKXGGPZAXYI7VKOSDEMJFO3I2QLLIF4CNYGGM3XBQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: <nil> Jul 26 23:32:01.121129125 (1h40m16.902062519s) [noise=true , address=79.127.205.234:4605 , piece_id=MJRU5HAPBF6ORRB4Z6FFWSI4KNMPIOOGRQMYGRSQOOUBYYP3AKUQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: <nil> Jul 26 23:32:01.397274599 (1h40m17.178207973s) [noise=true , address=79.127.219.47:4048 , piece_id=RB3ABLIP67YOFAHF2CDS2STPDURXAJZG2FP5GO2K2FBBS4B6DKNQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: <nil> Jul 26 23:32:02.379368187 (1h40m18.160301558s) [noise=true , address=45.91.116.9:4277 , piece_id=OX6EYFNEJCSQD7RDYEF7GDHKAAUNHN237EKYHPXQ465UW4YISCOA, seg_pos={0 0}, upload=1537]: Done uploading piece. err: <nil> Jul 26 23:32:02.738944034 (1h40m18.519877404s) [noise=true , address=156.146.43.225:4448 , piece_id=WCQMBCF2CUOSVQASPGY6THK2PACIDKPLLHAEHIINOOI66IOCY76Q, seg_pos={0 0}, upload=1537]: Done uploading piece. err: <nil> Jul 26 23:32:04.607773367 (1h40m20.388706752s) [noise=true , address=64.49.13.140:4508 , piece_id=EIHQTWDL72BNUCHSLGILXSZRREXXXBBRIUC3L5S3SZ6I37UYSFAQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 23:32:33.205246417 (1h40m48.986179795s) [noise=true , address=79.127.201.215:4344 , piece_id=XIKKXCYS46JFDZW7RMNC2HIVNC3YWB6THQ2TJAMNVJ4ULFCHTCKQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: <nil> Jul 26 23:33:00.428477210 (1h41m16.20941059s) [noise=true , address=104.36.82.46:4496 , piece_id=373IDZNHVH6QHAY6KZDDEFO2IDB4FFHCNFF7KXHDF6AOV366PHUQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: write tcp 159.223.143.39:55896->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:00.428848850 (1h41m16.209782227s) [noise=true , address=104.36.82.56:4230 , piece_id=5LBVPKYLT652M2K4FIDNJGIHU3R7Z4FW3UPGITE5RZCNQMOHGZJQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:12S5FyCphEYSuT8Bz2aZhiRkm6PYDnhD8NmdMQuXGscbMfaaeis, address:104.36.82.56:4230): protocol: rpc: dial tcp 104.36.82.56:4230: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:00.428934177 (1h41m16.209867545s) [noise=true , address=104.36.82.54:4167 , piece_id=PYTSSV6KF2RDQGCSAA26LFU5CM7FERKLMIHNBNAUUXJYV54F47GA, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ, address:104.36.82.54:4167): protocol: rpc: dial tcp 104.36.82.54:4167: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:00.429006265 (1h41m16.209939628s) [noise=true , address=104.36.82.55:4199 , piece_id=UIUKCUJY3RIV3KNTAOYZPPOUCNKJKND3BBQSNHW2D2IN5BANTZ6A, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:1mCH3fNEMd64WztUYYRCE2KUdJgLxWGoPUfEe4KaF34YhUxUXx, address:104.36.82.55:4199): protocol: rpc: dial tcp 104.36.82.55:4199: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:00.429159614 (1h41m16.210092991s) [noise=true , address=77.90.31.18:4831 , piece_id=7AGKK7XRXGSRCDVU3YJGS64OVZ6NRQWUSOWGKGKSA4F3DPKUUKUQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:1NenCLQMH3bdNj59r7KjQdMmwqe96gtBJdbj3faWuM2Ym3BDbA, address:77.90.31.18:4831): protocol: rpc: dial tcp 77.90.31.18:4831: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:00.429222931 (1h41m16.210156312s) [noise=true , address=104.36.82.52:4118 , piece_id=ANUDW6CVC52AZ6RA4ULRA46UIMC7L5PV4YKOTBSRQMMZAYSOOT4Q, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:128w37UHxJB9R49iwmHbATt3bXUb1FMTQJ3Wjka7So7eqKWg4iF, address:104.36.82.52:4118): protocol: rpc: dial tcp 104.36.82.52:4118: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:00.429250400 (1h41m16.210183782s) [noise=true , address=104.36.82.47:4532 , piece_id=G7V3LGUEFMNTJ5KRDHH6J5Y2VSAPJ6GFSR7Q7VNOBK5GXWJXGU7Q, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:15buPXDh9xXPzcL5Z7q2XZTQujajVRBp97VnPp5ngUDQL26AbL, address:104.36.82.47:4532): protocol: rpc: dial tcp 104.36.82.47:4532: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:00.429284001 (1h41m16.210217369s) [noise=true , address=104.36.82.10:4084 , piece_id=YJBAKYBYGA7ZNSKTKZHSPLXDBULTRQNXZ6P5FVQIBZ6E6QUDDIQQ, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:1mELnhMiRJ5GkUWYEggdxhCQ2JnDbT7qEMSiAuMQsF7BTnwP1u, address:104.36.82.10:4084): protocol: rpc: dial tcp 104.36.82.10:4084: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:00.429332105 (1h41m16.210265483s) [noise=true , address=104.36.82.53:4147 , piece_id=MHQE4C2UOOKWTOSIC3VT4DOG74BWAI225JW3SF7WSXLHZ6OR4H6A, seg_pos={0 0}, upload=1537]: Done uploading piece. err: ecclient: upload failed (node:12pg6fGWqRjS8Gf3zQgw8jbuytaCyMpJ2CAz4oRVHYdLAhttgnH, address:104.36.82.53:4147): protocol: write tcp 159.223.143.39:59358->104.36.82.53:4147: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:33:00.429377719 (1h41m16.210311112s) [seg_pos={0 0}, upload=1537]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 23:33:00.457182466 (1h41m16.238115845s) [upload=1537]: Done uploading segments... Jul 26 23:33:00.457215184 (1h41m16.238148563s) [upload=1537]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.03068491s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 31 out of 60 is failed (threshold=3) [.............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.032731352s FAIL (2025-07-26 23:32:33 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== .353122105 (1h40m19.134055493s) [seg_pos={0 0}, upload=1538]: Begin upload segment... Jul 26 23:32:03.353239411 (1h40m19.134172801s) [seg_pos={0 0}, upload=1538]: Done begin upload segment. Jul 26 23:32:03.353260643 (1h40m19.13419404s) [noise=true , address=104.36.82.10:4078 , piece_id=T4XORQPGNCLYENW54QLBPDUAHQ7S2X6HFKGFCEJC6QDSZBV772YA, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.353274424 (1h40m19.134207807s) [noise=true , address=45.91.116.4:4939 , piece_id=3KRX5PB2OSEIGFNO3UFIWLXTQJZU3KARWVYZRZK7BBI24KNWV2UA, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.353802423 (1h40m19.13473581s) [noise=true , address=45.91.116.6:4974 , piece_id=EZTTBOCNKNHFJEZVZ2ZXPIQEXX4533Q5APAUBJAJOUBJG7QY5N4A, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.353935259 (1h40m19.134868648s) [noise=true , address=45.91.116.5:4298 , piece_id=H27L56HRWWIQ4YAWTCTM5SDYAY5UNAWAZZ75RJG6KGHESMX7Y7ZQ, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.355345572 (1h40m19.136278956s) [noise=true , address=104.36.82.13:4187 , piece_id=XOXI42ULDEDSO3JTPCHVJHVIGMGURJ5L2WVZAAX2OUH3JOZPSQLA, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.355891971 (1h40m19.136825355s) [noise=true , address=45.91.116.6:4975 , piece_id=HUPUTYUEXVBVWVTOLCM2NUMWJDKYW7WYENA422YB2VZ3SJURVE7Q, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.356479709 (1h40m19.137413098s) [noise=true , address=104.36.82.2:4731 , piece_id=FO7NRU4GEDROJWQIZBGZAEQDKGUH6SIDGC3HYB6AK2F4TNZOUFOQ, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.357132302 (1h40m19.138065697s) [noise=true , address=104.36.82.8:4970 , piece_id=KDM65323RENUMUPSUXBHPGFE42X73HWDLF4E4IS64OJSHICOAKFQ, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.358048179 (1h40m19.138981566s) [noise=true , address=104.36.82.11:4115 , piece_id=3Z5T74QYNQUEAFUA3OXUOZPCZUVAXNBNHK4BW4Y3WLQEJQTKVYQA, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.360615159 (1h40m19.141548539s) [noise=true , address=104.36.82.14:4221 , piece_id=AYL23B7HRDJYXHDGKC5HSNRQRRX5WFOT7NJBRZIM4UXWG6P5PPGQ, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.361186918 (1h40m19.142120288s) [noise=true , address=104.36.82.12:4151 , piece_id=JMIOIHXIEI2FEI7S2YXLYH46S3WAOCGXRXSWP3SJF2ONODRUNYYA, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.361626661 (1h40m19.14256003s) [noise=true , address=104.36.82.7:4911 , piece_id=KQ7UHVHPW64JOA756ZGELUHK6OSUYLK6DRP4RDFPBIIZWDG2IY5A, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.362033254 (1h40m19.142966617s) [noise=true , address=104.36.82.9:4007 , piece_id=YJOAFWG4SZHNDTB6YC2XD4R3OFX4W6ZACUYEK637AG27IGDXXAXQ, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.362442429 (1h40m19.143375797s) [noise=true , address=104.36.82.15:4259 , piece_id=BQ7CKKHPF2K3I3IFZYQHOLXDZCVIOTQZJVTXTG4I6OHGSPFXGJIQ, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.362783491 (1h40m19.143716857s) [noise=true , address=104.36.82.6:4875 , piece_id=IDAQ4QP2KOYZMMFVDETP643A7AWXGXZNAQLW4QFC7YRWNMPJMXRQ, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.363421811 (1h40m19.144355189s) [noise=true , address=45.91.116.12:4225 , piece_id=LQ3QL7HRFD5V6DCAI4EHQUQLIUWTEURUSPNXIN5CULSQNV4HYKRA, seg_pos={0 0}, upload=1538]: Uploading piece... Jul 26 23:32:03.474840635 (1h40m19.255774015s) [upload=1538]: Next returned nil segment Jul 26 23:32:03.474863766 (1h40m19.255797155s) [upload=1538]: Waiting for error group managing segments... Jul 26 23:32:03.481844144 (1h40m19.262777526s) [noise=true , address=45.91.116.4:4939 , piece_id=3KRX5PB2OSEIGFNO3UFIWLXTQJZU3KARWVYZRZK7BBI24KNWV2UA, seg_pos={0 0}, upload=1538]: Done uploading piece. err: <nil> Jul 26 23:32:03.483000440 (1h40m19.26393382s) [noise=true , address=45.91.116.6:4974 , piece_id=EZTTBOCNKNHFJEZVZ2ZXPIQEXX4533Q5APAUBJAJOUBJG7QY5N4A, seg_pos={0 0}, upload=1538]: Done uploading piece. err: <nil> Jul 26 23:32:03.483951106 (1h40m19.264884478s) [noise=true , address=45.91.116.6:4975 , piece_id=HUPUTYUEXVBVWVTOLCM2NUMWJDKYW7WYENA422YB2VZ3SJURVE7Q, seg_pos={0 0}, upload=1538]: Done uploading piece. err: <nil> Jul 26 23:32:03.485508606 (1h40m19.266441987s) [noise=true , address=45.91.116.12:4225 , piece_id=LQ3QL7HRFD5V6DCAI4EHQUQLIUWTEURUSPNXIN5CULSQNV4HYKRA, seg_pos={0 0}, upload=1538]: Done uploading piece. err: <nil> Jul 26 23:32:03.782069594 (1h40m19.563002966s) [noise=true , address=45.91.116.5:4298 , piece_id=H27L56HRWWIQ4YAWTCTM5SDYAY5UNAWAZZ75RJG6KGHESMX7Y7ZQ, seg_pos={0 0}, upload=1538]: Done uploading piece. err: <nil> Jul 26 23:32:33.266246032 (1h40m49.047179409s) [noise=true , address=104.36.82.11:4115 , piece_id=3Z5T74QYNQUEAFUA3OXUOZPCZUVAXNBNHK4BW4Y3WLQEJQTKVYQA, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266289037 (1h40m49.047222415s) [noise=true , address=104.36.82.8:4970 , piece_id=KDM65323RENUMUPSUXBHPGFE42X73HWDLF4E4IS64OJSHICOAKFQ, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: rpc: dial tcp 104.36.82.8:4970: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266337416 (1h40m49.047270796s) [noise=true , address=104.36.82.14:4221 , piece_id=AYL23B7HRDJYXHDGKC5HSNRQRRX5WFOT7NJBRZIM4UXWG6P5PPGQ, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: rpc: dial tcp 104.36.82.14:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266394330 (1h40m49.047327691s) [noise=true , address=104.36.82.6:4875 , piece_id=IDAQ4QP2KOYZMMFVDETP643A7AWXGXZNAQLW4QFC7YRWNMPJMXRQ, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266452241 (1h40m49.047385618s) [noise=true , address=104.36.82.13:4187 , piece_id=XOXI42ULDEDSO3JTPCHVJHVIGMGURJ5L2WVZAAX2OUH3JOZPSQLA, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266476946 (1h40m49.04741031s) [noise=true , address=104.36.82.10:4078 , piece_id=T4XORQPGNCLYENW54QLBPDUAHQ7S2X6HFKGFCEJC6QDSZBV772YA, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266519005 (1h40m49.047452381s) [noise=true , address=104.36.82.2:4731 , piece_id=FO7NRU4GEDROJWQIZBGZAEQDKGUH6SIDGC3HYB6AK2F4TNZOUFOQ, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266546742 (1h40m49.047480118s) [noise=true , address=104.36.82.12:4151 , piece_id=JMIOIHXIEI2FEI7S2YXLYH46S3WAOCGXRXSWP3SJF2ONODRUNYYA, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266605231 (1h40m49.047538597s) [noise=true , address=104.36.82.7:4911 , piece_id=KQ7UHVHPW64JOA756ZGELUHK6OSUYLK6DRP4RDFPBIIZWDG2IY5A, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: write tcp 159.223.143.39:53120->104.36.82.7:4911: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266616554 (1h40m49.047549915s) [noise=true , address=104.36.82.15:4259 , piece_id=BQ7CKKHPF2K3I3IFZYQHOLXDZCVIOTQZJVTXTG4I6OHGSPFXGJIQ, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 159.223.143.39:39390->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266627022 (1h40m49.047560384s) [noise=true , address=104.36.82.9:4007 , piece_id=YJOAFWG4SZHNDTB6YC2XD4R3OFX4W6ZACUYEK637AG27IGDXXAXQ, seg_pos={0 0}, upload=1538]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:33.266648756 (1h40m49.047582138s) [seg_pos={0 0}, upload=1538]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:32:33.296633567 (1h40m49.077566933s) [upload=1538]: Done uploading segments... Jul 26 23:32:33.296666311 (1h40m49.077599697s) [upload=1538]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.032764262s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [...........................XXX.X....XX.X.X.X.......XXX...XX.] duration: 662.936088ms FAIL (2025-07-26 23:32:18 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 662.991151ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.....X.X..........XX...X...X..XX.X...X..X.......X...X....XX.] duration: 13.962058784s FAIL (2025-07-26 23:32:18 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 13.962087166s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 33 out of 60 is failed (threshold=3) [.....................X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 42.537929ms FAIL (2025-07-26 23:32:17 UTC) orbiter warning: uplink: too many requests Duration: 42.563919ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [............................XXX.X....XX.X.X.X.......XXX...XX] duration: 30.001015671s FAIL (2025-07-26 23:32:13 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001153328s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 32 out of 60 is failed (threshold=3) [......................X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.047768683s FAIL (2025-07-26 23:32:08 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.047800604s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [......X.X..........XX...X...X..XX.X...X..X.......X...X....XX] duration: 1m0.00256466s FAIL (2025-07-26 23:32:04 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.002588778s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 30 out of 60 is failed (threshold=3) [..............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.04311148s FAIL (2025-07-26 23:32:01 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== cessful: 12 optimal: 13 errs: <nil> Jul 26 23:30:54.744877179 (1h39m10.525810546s) [upload=1514]: Done uploading segments... Jul 26 23:30:54.744907487 (1h39m10.525840853s) [upload=1514]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 23:31:31.576458494 (1h39m47.357391884s) [upload=1529]: Starting upload Jul 26 23:31:31.576597271 (1h39m47.357530637s) [upload=1529]: Done starting upload Jul 26 23:31:31.576744624 (1h39m47.357677993s) [upload=1529]: Uploading segments... Jul 26 23:31:31.577178443 (1h39m47.358111832s) [upload=1529]: Got next segment. Inline: false Jul 26 23:31:31.577252019 (1h39m47.358185384s) [upload=1529]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:31:31.577260482 (1h39m47.358193867s) [upload=1529]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:31:31.608967209 (1h39m47.38990059s) [seg_pos={0 0}, upload=1529]: Begin upload segment... Jul 26 23:31:31.609041313 (1h39m47.389974675s) [seg_pos={0 0}, upload=1529]: Done begin upload segment. Jul 26 23:31:31.609054081 (1h39m47.389987471s) [noise=true , address=77.90.31.15:4698 , piece_id=7ZYN4RXVVEI6BS2JW4IVER7QNYF6JV6SZY3JREKKZUYBNVHQIR6Q, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.609065958 (1h39m47.389999343s) [noise=true , address=77.90.31.18:4805 , piece_id=3Q4F7FHNVURJIXDHTXU55ACLQENASWHGE4RSY744USTVD5DRXBUA, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.609564670 (1h39m47.390498058s) [noise=true , address=104.36.82.12:4149 , piece_id=PTPZSM423OKVFTTZBXYZBKYLEU72AQY55QBL2SE5CRRH7QKYHFDA, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.609594335 (1h39m47.390527718s) [noise=true , address=45.91.116.9:4263 , piece_id=BWHE6KLWXYR4G44A2U7WZV734XZFMIMHOUDNBVAZ4WS3XJI3XGZQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.610031867 (1h39m47.390965248s) [noise=true , address=104.36.82.7:4912 , piece_id=3Y5QPXF2KO3RSR35UGUR5JIMNRFNUJ4FEHPE2I4QD4HLDGVEGTRQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.610254432 (1h39m47.391187817s) [noise=true , address=104.36.82.15:4259 , piece_id=GEZGFRDTU2DM4IPPUIRYSGIRKBUWY5O3AZR4SFKDL3CA5OKYVHYQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.610538752 (1h39m47.39147214s) [noise=true , address=104.36.82.13:4185 , piece_id=BS2WXJ2N6ROPR26T3FH77UMGOSJEJ2T2KQV3Y7CV2VU4MDNYWQXQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.610641632 (1h39m47.391575018s) [noise=true , address=45.91.116.6:4974 , piece_id=JFAOMJOYSEPVXY2L37JHGXH4QFGW25WFJWU7SGKAF74FUVU7VI6A, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.611015304 (1h39m47.391948687s) [noise=true , address=45.91.116.13:4154 , piece_id=WIU4D4KNWUYTJDU67XUI5NL6CK6R45N7PCZN3XRGZG4NNIKERMQA, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.611768742 (1h39m47.392702129s) [noise=true , address=45.91.116.11:4119 , piece_id=SWSMSBOT3LYILWGTJFMJ6CAI57KBJFMMPPJO5CUUC7AIIAIHFTCQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.612180607 (1h39m47.393113995s) [noise=true , address=104.36.82.5:4839 , piece_id=U2GI3NFFYQMFJ7BQKUUD4AYJ2IRSUBNBOLYXUZMD7UPUBJYOFFKQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.612571320 (1h39m47.393504708s) [noise=true , address=104.36.82.11:4115 , piece_id=WFALXWKWHDQXLCDJRMVMTHMGWBKM6XSNU5K6T7EXVV72NVVPWJMQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.613078814 (1h39m47.394012201s) [noise=true , address=45.91.116.14:4189 , piece_id=6CIYF3QPA5LTUKN7Q2HCRX3HBU52OYX23K5ILWFG5VYVHQPJXFBQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.613860849 (1h39m47.394794237s) [noise=true , address=77.90.31.19:4841 , piece_id=WFVZIDNWXKGUWXSVHVFKNWJYVEDIYRGJFS5YDGNKZD6744XAU7MQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.614410419 (1h39m47.395343811s) [noise=true , address=45.91.116.5:4298 , piece_id=H3N45NWUCS2TAAUTGN5O342MN5VTWQM4V4SM5FPZ2QEBY5ZCTUIA, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.614924411 (1h39m47.395857799s) [noise=true , address=45.91.116.10:4081 , piece_id=7IWRBV5ZTO2C5CVTQCRCTTXXPJNPXL2KQWAJJL4TLGO76R2N3ZRQ, seg_pos={0 0}, upload=1529]: Uploading piece... Jul 26 23:31:31.724500405 (1h39m47.505433795s) [upload=1529]: Next returned nil segment Jul 26 23:31:31.724525907 (1h39m47.5054593s) [upload=1529]: Waiting for error group managing segments... Jul 26 23:31:31.733385014 (1h39m47.514318396s) [noise=true , address=45.91.116.13:4154 , piece_id=WIU4D4KNWUYTJDU67XUI5NL6CK6R45N7PCZN3XRGZG4NNIKERMQA, seg_pos={0 0}, upload=1529]: Done uploading piece. err: <nil> Jul 26 23:31:31.736702712 (1h39m47.517636096s) [noise=true , address=45.91.116.10:4081 , piece_id=7IWRBV5ZTO2C5CVTQCRCTTXXPJNPXL2KQWAJJL4TLGO76R2N3ZRQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: <nil> Jul 26 23:31:31.736898929 (1h39m47.5178323s) [noise=true , address=45.91.116.6:4974 , piece_id=JFAOMJOYSEPVXY2L37JHGXH4QFGW25WFJWU7SGKAF74FUVU7VI6A, seg_pos={0 0}, upload=1529]: Done uploading piece. err: <nil> Jul 26 23:31:31.746132211 (1h39m47.52706558s) [noise=true , address=77.90.31.15:4698 , piece_id=7ZYN4RXVVEI6BS2JW4IVER7QNYF6JV6SZY3JREKKZUYBNVHQIR6Q, seg_pos={0 0}, upload=1529]: Done uploading piece. err: <nil> Jul 26 23:31:31.781376503 (1h39m47.562309873s) [noise=true , address=45.91.116.11:4119 , piece_id=SWSMSBOT3LYILWGTJFMJ6CAI57KBJFMMPPJO5CUUC7AIIAIHFTCQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: <nil> Jul 26 23:31:31.826326868 (1h39m47.607260238s) [noise=true , address=45.91.116.14:4189 , piece_id=6CIYF3QPA5LTUKN7Q2HCRX3HBU52OYX23K5ILWFG5VYVHQPJXFBQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: <nil> Jul 26 23:31:31.966957645 (1h39m47.747891023s) [noise=true , address=45.91.116.5:4298 , piece_id=H3N45NWUCS2TAAUTGN5O342MN5VTWQM4V4SM5FPZ2QEBY5ZCTUIA, seg_pos={0 0}, upload=1529]: Done uploading piece. err: <nil> Jul 26 23:31:32.179086815 (1h39m47.960020198s) [noise=true , address=45.91.116.9:4263 , piece_id=BWHE6KLWXYR4G44A2U7WZV734XZFMIMHOUDNBVAZ4WS3XJI3XGZQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: <nil> Jul 26 23:31:33.122222841 (1h39m48.90315622s) [noise=true , address=77.90.31.19:4841 , piece_id=WFVZIDNWXKGUWXSVHVFKNWJYVEDIYRGJFS5YDGNKZD6744XAU7MQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: <nil> Jul 26 23:32:01.514610852 (1h40m17.295544215s) [noise=true , address=104.36.82.11:4115 , piece_id=WFALXWKWHDQXLCDJRMVMTHMGWBKM6XSNU5K6T7EXVV72NVVPWJMQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:56142->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:01.514704470 (1h40m17.295637842s) [noise=true , address=104.36.82.5:4839 , piece_id=U2GI3NFFYQMFJ7BQKUUD4AYJ2IRSUBNBOLYXUZMD7UPUBJYOFFKQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:50444->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:01.514767860 (1h40m17.295701222s) [noise=true , address=104.36.82.7:4912 , piece_id=3Y5QPXF2KO3RSR35UGUR5JIMNRFNUJ4FEHPE2I4QD4HLDGVEGTRQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:01.514953540 (1h40m17.295886921s) [noise=true , address=104.36.82.12:4149 , piece_id=PTPZSM423OKVFTTZBXYZBKYLEU72AQY55QBL2SE5CRRH7QKYHFDA, seg_pos={0 0}, upload=1529]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:01.515171579 (1h40m17.296104942s) [noise=true , address=104.36.82.15:4259 , piece_id=GEZGFRDTU2DM4IPPUIRYSGIRKBUWY5O3AZR4SFKDL3CA5OKYVHYQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:01.515298907 (1h40m17.296232272s) [noise=true , address=104.36.82.13:4185 , piece_id=BS2WXJ2N6ROPR26T3FH77UMGOSJEJ2T2KQV3Y7CV2VU4MDNYWQXQ, seg_pos={0 0}, upload=1529]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:01.515353693 (1h40m17.296287056s) [noise=true , address=77.90.31.18:4805 , piece_id=3Q4F7FHNVURJIXDHTXU55ACLQENASWHGE4RSY744USTVD5DRXBUA, seg_pos={0 0}, upload=1529]: Done uploading piece. err: ecclient: upload failed (node:12ExMgyeH6s1VMkWatpEqPQwMhX4aGJeK6AWn98kD4f6fQV7rcT, address:77.90.31.18:4805): protocol: read tcp 159.223.143.39:47822->77.90.31.18:4805: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:32:01.515375244 (1h40m17.296308622s) [seg_pos={0 0}, upload=1529]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:32:01.550559865 (1h40m17.331493233s) [upload=1529]: Done uploading segments... Jul 26 23:32:01.550603266 (1h40m17.331536643s) [upload=1529]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.04319252s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.025842406s FAIL (2025-07-26 23:31:53 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== eces (needed at least 14 but got 12) Jul 26 23:30:53.234981356 (1h39m9.015914735s) [upload=1523]: Starting upload Jul 26 23:30:53.235103174 (1h39m9.01603656s) [upload=1523]: Done starting upload Jul 26 23:30:53.235194854 (1h39m9.016128219s) [upload=1523]: Uploading segments... Jul 26 23:30:53.235479853 (1h39m9.016413231s) [upload=1523]: Got next segment. Inline: false Jul 26 23:30:53.235544266 (1h39m9.016477645s) [upload=1523]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:30:53.235556593 (1h39m9.016489969s) [upload=1523]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:30:53.280130951 (1h39m9.061064336s) [seg_pos={0 0}, upload=1523]: Begin upload segment... Jul 26 23:30:53.280213847 (1h39m9.061147232s) [seg_pos={0 0}, upload=1523]: Done begin upload segment. Jul 26 23:30:53.280505427 (1h39m9.061438799s) [noise=true , address=104.36.82.20:4448 , piece_id=JZ74YDQEPTPKI3C4LP5NUUXKZ47DPMYCJX6LHN7ZPLZEAGC37KVA, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.280565590 (1h39m9.061498972s) [noise=true , address=79.127.201.215:4355 , piece_id=MUXZIWTKSFNHRHCUALO7AD4ZJLZLUSDTVR5D7YVHLGU7AQZ2ILDQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.282010791 (1h39m9.062944305s) [noise=true , address=77.90.31.16:4736 , piece_id=JUC62XSYGC37MYPVAVIBAMI3EAUJ3QZR6RMEPK5S3ZL7DTPMED7Q, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.282579452 (1h39m9.063512816s) [noise=true , address=79.127.219.51:4210 , piece_id=IVKGFJ6IUIX36ADNH5OTAIUPZ6GZPQBOPZOQALVRZZ4R6MZBPWSQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.283594557 (1h39m9.064527926s) [noise=true , address=79.127.201.215:4354 , piece_id=BGSFXPMSS6PYPGMWBUFTHBDPKJAW3AH4SN42Z7QQLBWGI5I2PPZQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.283810763 (1h39m9.064744138s) [noise=true , address=79.127.205.246:4287 , piece_id=WJUMJZLHONESPZT3Z2CLH57OL4FCCZRV74NB5SAE237OI3W64DRA, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.284389366 (1h39m9.065322733s) [noise=true , address=64.49.13.140:4508 , piece_id=N3VC4AINDPE7PZT75AF7DHZTKYUUDPBRNE4LCOIFXN7HX5SGOQXQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.284562838 (1h39m9.06549621s) [noise=true , address=79.127.205.235:4675 , piece_id=2AKE26X7XTKCBXIXQF2ZQBRCL2CVGEKZMLDA2TRZFUITN54ZJOPA, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.285085172 (1h39m9.066018541s) [noise=true , address=45.91.116.14:4194 , piece_id=5G3IQOILJAPO7KLHLNYPYBO3ZLNDI52643MTPZQN3GIC3W4DD4IQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.285422117 (1h39m9.066355486s) [noise=true , address=104.36.82.55:4221 , piece_id=ALTQAIV6CSRXASRCS3UNS36ASMQTVZUTO4O2OQXAXSU5XWKNPKIQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.285757431 (1h39m9.0666908s) [noise=true , address=79.127.205.234:4609 , piece_id=JGGVMCED4XAQYTNYE6TTXSZG72GOOGW73PW35FHTCKLT4NQFRFIQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.286067371 (1h39m9.067000742s) [noise=true , address=104.36.82.53:4128 , piece_id=TLSTW52JLUKIKNS67NJCB2EH4V3Y24QJ6OV6VB5BU4WUXWWFJN7Q, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.286386974 (1h39m9.067320345s) [noise=true , address=79.127.205.245:4225 , piece_id=KA22FTY3LYQSL4HNXBIELAD37LEH55UVHYGRQ2T3KTJB7KC5HUQQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.286669012 (1h39m9.06760238s) [noise=true , address=79.127.219.47:4048 , piece_id=MTP7ON4CLWIU6JGKVINATUPBKBQPC7FSRNWNVSBYDYRB3RQDYBYA, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.287043911 (1h39m9.067977281s) [noise=true , address=77.90.31.15:4724 , piece_id=VSNW47ZUPIY3ZVZYDEVDQBT6ZLCOP4PJ7HLHOI5ZFFI4MMYF2BHQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.287479008 (1h39m9.068412375s) [noise=true , address=104.36.82.54:4174 , piece_id=GWJNTC5354WDUCMIS76YECL7DNPHTJBYWWWFWN46TGFZPIHTFMPA, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.287787298 (1h39m9.068720671s) [noise=true , address=104.36.82.47:4552 , piece_id=SWQABTTDRTVIBCWANZSIKWOVTY6NWF63RUQAZZAARMGDCBYJO7UQ, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.288151713 (1h39m9.069085093s) [noise=true , address=79.127.201.215:4349 , piece_id=2F6GEWJLCCEAW2XC4KP6U2QVT6LSIYBILTRUJUNRKLR4BVFJSMVA, seg_pos={0 0}, upload=1523]: Uploading piece... Jul 26 23:30:53.372316009 (1h39m9.153249371s) [upload=1523]: Next returned nil segment Jul 26 23:30:53.372341457 (1h39m9.153274833s) [upload=1523]: Waiting for error group managing segments... Jul 26 23:30:53.413241358 (1h39m9.194174739s) [noise=true , address=77.90.31.15:4724 , piece_id=VSNW47ZUPIY3ZVZYDEVDQBT6ZLCOP4PJ7HLHOI5ZFFI4MMYF2BHQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:30:53.626257370 (1h39m9.407190733s) [noise=true , address=79.127.219.51:4210 , piece_id=IVKGFJ6IUIX36ADNH5OTAIUPZ6GZPQBOPZOQALVRZZ4R6MZBPWSQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:30:53.874888347 (1h39m9.655821716s) [noise=true , address=79.127.205.235:4675 , piece_id=2AKE26X7XTKCBXIXQF2ZQBRCL2CVGEKZMLDA2TRZFUITN54ZJOPA, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:30:53.890001423 (1h39m9.670934792s) [noise=true , address=79.127.205.245:4225 , piece_id=KA22FTY3LYQSL4HNXBIELAD37LEH55UVHYGRQ2T3KTJB7KC5HUQQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:30:53.907214248 (1h39m9.688147614s) [noise=true , address=79.127.219.47:4048 , piece_id=MTP7ON4CLWIU6JGKVINATUPBKBQPC7FSRNWNVSBYDYRB3RQDYBYA, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:30:53.949228336 (1h39m9.73016172s) [noise=true , address=79.127.205.246:4287 , piece_id=WJUMJZLHONESPZT3Z2CLH57OL4FCCZRV74NB5SAE237OI3W64DRA, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:30:54.824171521 (1h39m10.60510489s) [noise=true , address=79.127.205.234:4609 , piece_id=JGGVMCED4XAQYTNYE6TTXSZG72GOOGW73PW35FHTCKLT4NQFRFIQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:30:58.994629630 (1h39m14.775562993s) [noise=true , address=45.91.116.14:4194 , piece_id=5G3IQOILJAPO7KLHLNYPYBO3ZLNDI52643MTPZQN3GIC3W4DD4IQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:31:00.298638954 (1h39m16.079572325s) [noise=true , address=64.49.13.140:4508 , piece_id=N3VC4AINDPE7PZT75AF7DHZTKYUUDPBRNE4LCOIFXN7HX5SGOQXQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:31:09.244670085 (1h39m25.025603452s) [noise=true , address=79.127.201.215:4355 , piece_id=MUXZIWTKSFNHRHCUALO7AD4ZJLZLUSDTVR5D7YVHLGU7AQZ2ILDQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:31:15.992987059 (1h39m31.773920439s) [noise=true , address=77.90.31.16:4736 , piece_id=JUC62XSYGC37MYPVAVIBAMI3EAUJ3QZR6RMEPK5S3ZL7DTPMED7Q, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:31:30.443778609 (1h39m46.224711971s) [noise=true , address=79.127.201.215:4354 , piece_id=BGSFXPMSS6PYPGMWBUFTHBDPKJAW3AH4SN42Z7QQLBWGI5I2PPZQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:31:46.506777124 (1h40m2.287710501s) [noise=true , address=79.127.201.215:4349 , piece_id=2F6GEWJLCCEAW2XC4KP6U2QVT6LSIYBILTRUJUNRKLR4BVFJSMVA, seg_pos={0 0}, upload=1523]: Done uploading piece. err: <nil> Jul 26 23:31:53.198614381 (1h40m8.979547762s) [noise=true , address=104.36.82.55:4221 , piece_id=ALTQAIV6CSRXASRCS3UNS36ASMQTVZUTO4O2OQXAXSU5XWKNPKIQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: ecclient: upload failed (node:1ssFiXk5oKTgnVMJ5TBJuSXuD1D2C46gT2hcY6yaDxf8fL1cjm, address:104.36.82.55:4221): protocol: rpc: dial tcp 104.36.82.55:4221: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.198696025 (1h40m8.979629391s) [noise=true , address=104.36.82.54:4174 , piece_id=GWJNTC5354WDUCMIS76YECL7DNPHTJBYWWWFWN46TGFZPIHTFMPA, seg_pos={0 0}, upload=1523]: Done uploading piece. err: ecclient: upload failed (node:1293SNDGTKNrYrKDuB5p2VpHWo7ANpQsVKVGa97jXVaGP83Pw51, address:104.36.82.54:4174): protocol: rpc: dial tcp 104.36.82.54:4174: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.198755921 (1h40m8.979689287s) [noise=true , address=104.36.82.47:4552 , piece_id=SWQABTTDRTVIBCWANZSIKWOVTY6NWF63RUQAZZAARMGDCBYJO7UQ, seg_pos={0 0}, upload=1523]: Done uploading piece. err: ecclient: upload failed (node:1hoXgBFoGKKknrobbqfArxNhrsiUHKiiVZGWCnvxDSgBcae8gw, address:104.36.82.47:4552): protocol: rpc: dial tcp 104.36.82.47:4552: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.198790524 (1h40m8.979723901s) [noise=true , address=104.36.82.20:4448 , piece_id=JZ74YDQEPTPKI3C4LP5NUUXKZ47DPMYCJX6LHN7ZPLZEAGC37KVA, seg_pos={0 0}, upload=1523]: Done uploading piece. err: ecclient: upload failed (node:12vbYQVrjLwiHTMGsAGAcjze8KeVQUqemqGiVTWF63Xq6F7ABUp, address:104.36.82.20:4448): protocol: rpc: dial tcp 104.36.82.20:4448: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:31:53.198937385 (1h40m8.97987077s) [noise=true , address=104.36.82.53:4128 , piece_id=TLSTW52JLUKIKNS67NJCB2EH4V3Y24QJ6OV6VB5BU4WUXWWFJN7Q, seg_pos={0 0}, upload=1523]: Done uploading piece. err: ecclient: upload failed (node:125ZMc6ed9qgBRtBC5N4zNvGonip3GNAEBPt3auF6dBGteX46Pf, address:104.36.82.53:4128): protocol: write tcp 159.223.143.39:35398->104.36.82.53:4128: use of closed network connection; conn closed: op Jul 26 23:31:53.199013779 (1h40m8.979947202s) [seg_pos={0 0}, upload=1523]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 23:31:53.222144113 (1h40m9.003077491s) [upload=1523]: Done uploading segments... Jul 26 23:31:53.222175757 (1h40m9.003109158s) [upload=1523]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.025865114s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [.............................XXX.X....XX.X.X.X.......XXX...X] duration: 30.001016143s FAIL (2025-07-26 23:31:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001104709s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 31 out of 60 is failed (threshold=3) [.......................X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.046083261s FAIL (2025-07-26 23:31:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.046123384s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.......X.X..........XX...X...X..XX.X...X..X.......X...X....X] duration: 1m0.004400598s FAIL (2025-07-26 23:31:04 UTC) context deadline exceeded Duration: 1m0.004537097s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 29 out of 60 is failed (threshold=3) [...............................XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.026265582s FAIL (2025-07-26 23:30:54 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== load=1505]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:49066->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:14.055820922 (1h38m29.836754301s) [noise=true , address=104.36.82.5:4837 , piece_id=AUCQLQSGFAFE7HB3XVND52T35EEF4WAWMH654JLGT5XNCKKPB6WQ, seg_pos={0 0}, upload=1505]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 159.223.143.39:36558->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:14.055889162 (1h38m29.836822545s) [seg_pos={0 0}, upload=1505]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:30:14.088690384 (1h38m29.869623767s) [upload=1505]: Done uploading segments... Jul 26 23:30:14.088721855 (1h38m29.869655239s) [upload=1505]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Jul 26 23:30:24.751956304 (1h38m40.532889692s) [upload=1514]: Starting upload Jul 26 23:30:24.752208937 (1h38m40.533142326s) [upload=1514]: Done starting upload Jul 26 23:30:24.752610272 (1h38m40.533543655s) [upload=1514]: Uploading segments... Jul 26 23:30:24.752733968 (1h38m40.533667348s) [upload=1514]: Got next segment. Inline: false Jul 26 23:30:24.752787268 (1h38m40.53372065s) [upload=1514]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:30:24.752794341 (1h38m40.533727726s) [upload=1514]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:30:24.791595643 (1h38m40.572529018s) [seg_pos={0 0}, upload=1514]: Begin upload segment... Jul 26 23:30:24.791655519 (1h38m40.572588902s) [seg_pos={0 0}, upload=1514]: Done begin upload segment. Jul 26 23:30:24.791687905 (1h38m40.572621291s) [noise=true , address=77.90.31.21:4879 , piece_id=LOBQUKNC3GST3UEODOZPG6Z7OYHXHSI7APKZSTVW26IWBWFLGPMQ, seg_pos={0 0}, upload=1514]: Uploading piece... Jul 26 23:30:24.792447116 (1h38m40.573380503s) [noise=true , address=45.91.116.6:4973 , piece_id=U5NOQHIWCQMQG3QUVS3U3LH4FOLDMCJ35AQ66KX2DMWB6X4XLVSA, seg_pos={0 0}, upload=1514]: Uploading piece... Jul 26 23:30:24.792869467 (1h38m40.573802857s) [noise=true , address=45.91.116.7:4009 , piece_id=H7TAWXD2IZQQY2EFZE6SYLBHIPGSLIKIONT2A45KGJB35BAWEGDA, seg_pos={0 0}, upload=1514]: Uploading piece... Jul 26 23:30:24.793242427 (1h38m40.574175817s) [noise=true , address=77.90.31.11:4553 , piece_id=SNX47JTSPFGQ6VAF6W7OZAMVV5BA44NZKSIMUEO4NOITBCV3KPTA, seg_pos={0 0}, upload=1514]: Uploading piece... Jul 26 23:30:24.793651765 (1h38m40.574585146s) [noise=true , address=104.36.82.9:4007 , piece_id=QDXI5WATI6RF42C5C2ZADZWZWUKZS5GM2B4BR54MIPJAVR2ENH7Q, seg_pos={0 0}, upload=1514]: Uploading piece... Jul 26 23:30:24.794036396 (1h38m40.574969761s) [noise=true , address=104.36.82.4:4803 , piece_id=H2NIWX26JP4V6BXM2OJNT3R5PDWYHQMM3VBZKT22GPZBM3Q3HWIQ, seg_pos={0 0}, upload=1514]: Uploading piece... Jul 26 23:30:24.794444581 (1h38m40.575377966s) [noise=true , address=77.90.31.21:4877 , piece_id=5SKEYPYQ6LNVIO4TD5REV7YZUCDJ56XHBYRCGJOCGOON6MQUJJ6A, seg_pos={0 0}, upload=1514]: Uploading piece... Jul 26 23:30:24.794889574 (1h38m40.575822957s) [noise=true , address=45.91.116.12:4227 , piece_id=MXS3RN6AMKC4WVY7OUCGHZ3ALQ3GPXPDPIUTHVP4W4ETALBGT2JQ, seg_pos={0 0}, upload=1514]: Uploading piece... Jul 26 23:30:24.795291301 (1h38m40.576224692s) [noise=true , address=45.91.116.14:4189 , piece_id=HYCSPRLAVBYMMOKJI3RERID7RJNKUM3TGCT36UZDJXOQPR6NAOHA, seg_pos={0 0}, upload=1514]: Uploading piece... Jul 26 23:30:24.795718094 (1h38m40.576651483s) [noise=true , address=45.91.116.13:4155 , piece_id=XCCW6WODMILNOUUBP47OF7RJH7IFQBVB4GQP52KO3CFYJUIZ47HQ, seg_pos={0 0}, upload=1514]: Uploading piece... Jul 26 23:30:24.796123754 (1h38m40.577057144s) [noise=true , address=77.90.31.15:4697 , piece_id=IECG7NGQJ5DOW6F4Z4MX7WNJITXJYQQMKQORGBJ5FAUW5TKKG4ZA, seg_pos={0 0}, upload=1514]: Uploading piece... Jul 26 23:30:24.796532543 (1h38m40.577465931s) [noise=true , address=104.36.82.16:4293 , piece_id=FLRCYHVBWFGIMPKMKDWDGOWSRUWNWA6BPP66BRTCXIM56WOSG3EA, seg_pos={0 0}, upload=1514]: Uploading piece... Jul 26 23:30:24.796953116 (1h38m40.577886503s) [noise=true , address=45.91.116.11:4119 , piece_id=GAUFKLUDZL6QOMM6W5QAOIXRME2GPEWLQU7Y37VEFCZSDV7R6F6Q, seg_pos={0 0}, upload=1514]: Uploading piece... Jul 26 23:30:24.797323252 (1h38m40.578256637s) [noise=true , address=45.91.116.6:4975 , piece_id=5YIYXSTLK6XDRKKEVAT4C4HTNV2UCSEH6RK6MTCGLWE3OOL2YVCA, seg_pos={0 0}, upload=1514]: Uploading piece... Jul 26 23:30:24.797693404 (1h38m40.578626787s) [noise=true , address=45.91.116.12:4226 , piece_id=K5UBY7NESK7CIKGCHNOANQMDNVAABCGLW3263BO35LWDJBFTPZ5A, seg_pos={0 0}, upload=1514]: Uploading piece... Jul 26 23:30:24.798066042 (1h38m40.578999431s) [noise=true , address=45.91.116.7:4011 , piece_id=DRL4VG67DHS7YENIK7PIDJARJD4QGOBC5HBULFVL62M7T74A2KAA, seg_pos={0 0}, upload=1514]: Uploading piece... Jul 26 23:30:24.912267139 (1h38m40.693200508s) [upload=1514]: Next returned nil segment Jul 26 23:30:24.912289685 (1h38m40.693223071s) [upload=1514]: Waiting for error group managing segments... Jul 26 23:30:24.916598056 (1h38m40.697531434s) [noise=true , address=77.90.31.21:4877 , piece_id=5SKEYPYQ6LNVIO4TD5REV7YZUCDJ56XHBYRCGJOCGOON6MQUJJ6A, seg_pos={0 0}, upload=1514]: Done uploading piece. err: <nil> Jul 26 23:30:24.917554515 (1h38m40.698487898s) [noise=true , address=45.91.116.12:4226 , piece_id=K5UBY7NESK7CIKGCHNOANQMDNVAABCGLW3263BO35LWDJBFTPZ5A, seg_pos={0 0}, upload=1514]: Done uploading piece. err: <nil> Jul 26 23:30:24.918123883 (1h38m40.699057261s) [noise=true , address=45.91.116.13:4155 , piece_id=XCCW6WODMILNOUUBP47OF7RJH7IFQBVB4GQP52KO3CFYJUIZ47HQ, seg_pos={0 0}, upload=1514]: Done uploading piece. err: <nil> Jul 26 23:30:24.918539952 (1h38m40.699473335s) [noise=true , address=45.91.116.7:4009 , piece_id=H7TAWXD2IZQQY2EFZE6SYLBHIPGSLIKIONT2A45KGJB35BAWEGDA, seg_pos={0 0}, upload=1514]: Done uploading piece. err: <nil> Jul 26 23:30:24.923089616 (1h38m40.704022986s) [noise=true , address=45.91.116.6:4973 , piece_id=U5NOQHIWCQMQG3QUVS3U3LH4FOLDMCJ35AQ66KX2DMWB6X4XLVSA, seg_pos={0 0}, upload=1514]: Done uploading piece. err: <nil> Jul 26 23:30:24.923691575 (1h38m40.704624943s) [noise=true , address=45.91.116.7:4011 , piece_id=DRL4VG67DHS7YENIK7PIDJARJD4QGOBC5HBULFVL62M7T74A2KAA, seg_pos={0 0}, upload=1514]: Done uploading piece. err: <nil> Jul 26 23:30:24.924237229 (1h38m40.705170593s) [noise=true , address=45.91.116.12:4227 , piece_id=MXS3RN6AMKC4WVY7OUCGHZ3ALQ3GPXPDPIUTHVP4W4ETALBGT2JQ, seg_pos={0 0}, upload=1514]: Done uploading piece. err: <nil> Jul 26 23:30:24.924748977 (1h38m40.705682347s) [noise=true , address=77.90.31.21:4879 , piece_id=LOBQUKNC3GST3UEODOZPG6Z7OYHXHSI7APKZSTVW26IWBWFLGPMQ, seg_pos={0 0}, upload=1514]: Done uploading piece. err: <nil> Jul 26 23:30:24.925912354 (1h38m40.706845731s) [noise=true , address=45.91.116.6:4975 , piece_id=5YIYXSTLK6XDRKKEVAT4C4HTNV2UCSEH6RK6MTCGLWE3OOL2YVCA, seg_pos={0 0}, upload=1514]: Done uploading piece. err: <nil> Jul 26 23:30:24.926601316 (1h38m40.707534694s) [noise=true , address=77.90.31.15:4697 , piece_id=IECG7NGQJ5DOW6F4Z4MX7WNJITXJYQQMKQORGBJ5FAUW5TKKG4ZA, seg_pos={0 0}, upload=1514]: Done uploading piece. err: <nil> Jul 26 23:30:24.958479100 (1h38m40.739412472s) [noise=true , address=45.91.116.14:4189 , piece_id=HYCSPRLAVBYMMOKJI3RERID7RJNKUM3TGCT36UZDJXOQPR6NAOHA, seg_pos={0 0}, upload=1514]: Done uploading piece. err: <nil> Jul 26 23:30:24.978756005 (1h38m40.759689373s) [noise=true , address=45.91.116.11:4119 , piece_id=GAUFKLUDZL6QOMM6W5QAOIXRME2GPEWLQU7Y37VEFCZSDV7R6F6Q, seg_pos={0 0}, upload=1514]: Done uploading piece. err: <nil> Jul 26 23:30:54.720686136 (1h39m10.501619501s) [noise=true , address=77.90.31.11:4553 , piece_id=SNX47JTSPFGQ6VAF6W7OZAMVV5BA44NZKSIMUEO4NOITBCV3KPTA, seg_pos={0 0}, upload=1514]: Done uploading piece. err: ecclient: upload failed (node:1ULMQbw1PRksy5un9tNpwTQFgZoNsy9PhUn86Ny9itKAkesshz, address:77.90.31.11:4553): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:30:54.720733992 (1h39m10.501667377s) [noise=true , address=104.36.82.9:4007 , piece_id=QDXI5WATI6RF42C5C2ZADZWZWUKZS5GM2B4BR54MIPJAVR2ENH7Q, seg_pos={0 0}, upload=1514]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:54.720748702 (1h39m10.501682153s) [noise=true , address=104.36.82.4:4803 , piece_id=H2NIWX26JP4V6BXM2OJNT3R5PDWYHQMM3VBZKT22GPZBM3Q3HWIQ, seg_pos={0 0}, upload=1514]: Done uploading piece. err: ecclient: upload failed (node:12HWjkru3nFUa9hvA27sHqLEmRHSyXTV45phS5kMHKWM163M4H7, address:104.36.82.4:4803): protocol: rpc: dial tcp 104.36.82.4:4803: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:54.720988181 (1h39m10.501921548s) [noise=true , address=104.36.82.16:4293 , piece_id=FLRCYHVBWFGIMPKMKDWDGOWSRUWNWA6BPP66BRTCXIM56WOSG3EA, seg_pos={0 0}, upload=1514]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:30:54.721051575 (1h39m10.501984957s) [seg_pos={0 0}, upload=1514]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:30:54.744877179 (1h39m10.525810546s) [upload=1514]: Done uploading segments... Jul 26 23:30:54.744907487 (1h39m10.525840853s) [upload=1514]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.026293919s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [X..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.03019553s FAIL (2025-07-26 23:30:53 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 6 23:29:44.101793455 (1h37m59.882726831s) [upload=1492]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Jul 26 23:29:53.131461582 (1h38m8.912394948s) [upload=1507]: Starting upload Jul 26 23:29:53.131544792 (1h38m8.912478158s) [upload=1507]: Done starting upload Jul 26 23:29:53.131638375 (1h38m8.912571742s) [upload=1507]: Uploading segments... Jul 26 23:29:53.131949397 (1h38m8.912882785s) [upload=1507]: Got next segment. Inline: false Jul 26 23:29:53.132200967 (1h38m8.913134355s) [upload=1507]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:29:53.132212140 (1h38m8.913145529s) [upload=1507]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:29:53.171891100 (1h38m8.952824481s) [seg_pos={0 0}, upload=1507]: Begin upload segment... Jul 26 23:29:53.171949230 (1h38m8.952882608s) [seg_pos={0 0}, upload=1507]: Done begin upload segment. Jul 26 23:29:53.171979885 (1h38m8.952913269s) [noise=true , address=77.90.31.21:4904 , piece_id=CI3QUKOABTTDHJLQSVTYPE2VDQQWBFPBZYR22YIUA3A4CLEMZLSQ, seg_pos={0 0}, upload=1507]: Uploading piece... Jul 26 23:29:53.172152982 (1h38m8.953086363s) [noise=true , address=79.127.205.234:4615 , piece_id=IA3YON6NLKH5VJOOBFKLD342NHBSO43I7DZGCJSRZOQ4HEMF53NQ, seg_pos={0 0}, upload=1507]: Uploading piece... Jul 26 23:29:53.172581685 (1h38m8.953515075s) [noise=true , address=79.127.205.235:4690 , piece_id=HP3WR22FJ4YTQN4DGJA7DY6JDIKT3HV7ZEUU3WABOIWC5MDSUCMQ, seg_pos={0 0}, upload=1507]: Uploading piece... Jul 26 23:29:53.172788154 (1h38m8.953721537s) [noise=true , address=79.127.219.38:4007 , piece_id=IXHATSWUPN3WLHRO46FPJ5FJFKTD236XKTDWK6UH3VV5QMFDR53Q, seg_pos={0 0}, upload=1507]: Uploading piece... Jul 26 23:29:53.173012538 (1h38m8.953945925s) [noise=true , address=104.36.82.54:4168 , piece_id=IQXVK6FZFAL7QUV4FDMQSP33FSHU36SVDPOUHEFDPIEA27AAERCA, seg_pos={0 0}, upload=1507]: Uploading piece... Jul 26 23:29:53.173367560 (1h38m8.954300926s) [noise=true , address=77.90.31.10:4528 , piece_id=SGCCQE7MEKH22A32ZXKM7I3MIRYGXEEEGYGPMQCK4VCD4BAQIWSQ, seg_pos={0 0}, upload=1507]: Uploading piece... Jul 26 23:29:53.173451951 (1h38m8.954385334s) [noise=true , address=79.127.205.246:4284 , piece_id=RPG3LCGHJ4J4DCVQLGINYA6O4ZOKCZVNTM34MRB2BLGMBX3FSIAQ, seg_pos={0 0}, upload=1507]: Uploading piece... Jul 26 23:29:53.173842025 (1h38m8.954775414s) [noise=true , address=79.127.201.214:4321 , piece_id=N7PNTPBU3DTY27RSJWE2AFWBYWQKDIDGJC7QEKR363HN6YVAHYQA, seg_pos={0 0}, upload=1507]: Uploading piece... Jul 26 23:29:53.173865948 (1h38m8.954799336s) [noise=true , address=104.36.82.4:4822 , piece_id=2WDNUC4O3T7XQCSNX75ITSZ5GX2ZX2OAYQ5F5PQZYRSOYGEMQQTQ, seg_pos={0 0}, upload=1507]: Uploading piece... Jul 26 23:29:53.174253481 (1h38m8.955186869s) [noise=true , address=77.90.31.11:4577 , piece_id=JGVMAJUXDJNJC2F7EOSOQK7K7EZKSQEDO4DXVGY6QMZPXC6UW5KQ, seg_pos={0 0}, upload=1507]: Uploading piece... Jul 26 23:29:53.174385073 (1h38m8.955318461s) [noise=true , address=104.36.82.46:4485 , piece_id=GGP4SYW5Y2K5T6EIYZHVQOVHBWVNBE2X4VEKEYAKFNJZLCPWTZRA, seg_pos={0 0}, upload=1507]: Uploading piece... Jul 26 23:29:53.174652785 (1h38m8.955586166s) [noise=true , address=79.127.205.245:4220 , piece_id=KLGXQMINYSOPWTQF5UF3FU6WABLQTESOQBMYZVLSOP7ZBDMCDPSQ, seg_pos={0 0}, upload=1507]: Uploading piece... Jul 26 23:29:53.174892226 (1h38m8.95582561s) [noise=true , address=77.90.31.15:4730 , piece_id=DNHSS6LRRIXXBDI4OTOEN44EAW2FL2CYABIWSFC4ZZZIHBDHZAEQ, seg_pos={0 0}, upload=1507]: Uploading piece... Jul 26 23:29:53.175102847 (1h38m8.956036233s) [noise=true , address=104.36.82.47:4525 , piece_id=AOKK77ACINTXERF4ULGUJESEHELMZEJYH3SNIJ4UUNACL6F6TPEA, seg_pos={0 0}, upload=1507]: Uploading piece... Jul 26 23:29:53.175510776 (1h38m8.956444164s) [noise=true , address=64.49.13.138:4482 , piece_id=JLAHJEQK7WT7QETPXNLTAG24WJ6GYNLVBUFYCONH4JDDATM4HHFA, seg_pos={0 0}, upload=1507]: Uploading piece... Jul 26 23:29:53.175527062 (1h38m8.956460449s) [noise=true , address=77.90.31.16:4753 , piece_id=OVD27UMNYNVQBMKRLGFCBVJAI55BCIZ5QOKZN4CYDZA3YB4MEDZA, seg_pos={0 0}, upload=1507]: Uploading piece... Jul 26 23:29:53.175967031 (1h38m8.95690042s) [noise=true , address=64.49.13.137:4502 , piece_id=DEQ5MQJYAZP4HQGNOK6WW6P2GBEGW5NELYCPKIJOBQRVKZPVNP7Q, seg_pos={0 0}, upload=1507]: Uploading piece... Jul 26 23:29:53.178429135 (1h38m8.959362498s) [noise=true , address=104.36.82.42:4807 , piece_id=7OFW3XZRU3RAJM4Z2DOKXIQ45IFT74A2LMLZVI7HN5BZHCWZWF6Q, seg_pos={0 0}, upload=1507]: Uploading piece... Jul 26 23:29:53.267590155 (1h38m9.048523545s) [upload=1507]: Next returned nil segment Jul 26 23:29:53.267611867 (1h38m9.048545256s) [upload=1507]: Waiting for error group managing segments... Jul 26 23:29:53.318432466 (1h38m9.099365845s) [noise=true , address=77.90.31.21:4904 , piece_id=CI3QUKOABTTDHJLQSVTYPE2VDQQWBFPBZYR22YIUA3A4CLEMZLSQ, seg_pos={0 0}, upload=1507]: Done uploading piece. err: <nil> Jul 26 23:29:53.319711702 (1h38m9.100645087s) [noise=true , address=77.90.31.10:4528 , piece_id=SGCCQE7MEKH22A32ZXKM7I3MIRYGXEEEGYGPMQCK4VCD4BAQIWSQ, seg_pos={0 0}, upload=1507]: Done uploading piece. err: <nil> Jul 26 23:29:53.338682256 (1h38m9.119615639s) [noise=true , address=79.127.219.38:4007 , piece_id=IXHATSWUPN3WLHRO46FPJ5FJFKTD236XKTDWK6UH3VV5QMFDR53Q, seg_pos={0 0}, upload=1507]: Done uploading piece. err: <nil> Jul 26 23:29:53.735103381 (1h38m9.516036748s) [noise=true , address=79.127.205.235:4690 , piece_id=HP3WR22FJ4YTQN4DGJA7DY6JDIKT3HV7ZEUU3WABOIWC5MDSUCMQ, seg_pos={0 0}, upload=1507]: Done uploading piece. err: <nil> Jul 26 23:29:53.761136904 (1h38m9.542070287s) [noise=true , address=79.127.205.246:4284 , piece_id=RPG3LCGHJ4J4DCVQLGINYA6O4ZOKCZVNTM34MRB2BLGMBX3FSIAQ, seg_pos={0 0}, upload=1507]: Done uploading piece. err: <nil> Jul 26 23:29:53.766023324 (1h38m9.546956689s) [noise=true , address=79.127.205.245:4220 , piece_id=KLGXQMINYSOPWTQF5UF3FU6WABLQTESOQBMYZVLSOP7ZBDMCDPSQ, seg_pos={0 0}, upload=1507]: Done uploading piece. err: <nil> Jul 26 23:29:53.823657902 (1h38m9.604591284s) [noise=true , address=64.49.13.137:4502 , piece_id=DEQ5MQJYAZP4HQGNOK6WW6P2GBEGW5NELYCPKIJOBQRVKZPVNP7Q, seg_pos={0 0}, upload=1507]: Done uploading piece. err: <nil> Jul 26 23:29:53.898375693 (1h38m9.679309056s) [noise=true , address=64.49.13.138:4482 , piece_id=JLAHJEQK7WT7QETPXNLTAG24WJ6GYNLVBUFYCONH4JDDATM4HHFA, seg_pos={0 0}, upload=1507]: Done uploading piece. err: <nil> Jul 26 23:29:54.257438154 (1h38m10.038371548s) [noise=true , address=79.127.205.234:4615 , piece_id=IA3YON6NLKH5VJOOBFKLD342NHBSO43I7DZGCJSRZOQ4HEMF53NQ, seg_pos={0 0}, upload=1507]: Done uploading piece. err: <nil> Jul 26 23:29:56.206784854 (1h38m11.987718224s) [noise=true , address=79.127.201.214:4321 , piece_id=N7PNTPBU3DTY27RSJWE2AFWBYWQKDIDGJC7QEKR363HN6YVAHYQA, seg_pos={0 0}, upload=1507]: Done uploading piece. err: <nil> Jul 26 23:30:14.302687981 (1h38m30.08362136s) [noise=true , address=77.90.31.15:4730 , piece_id=DNHSS6LRRIXXBDI4OTOEN44EAW2FL2CYABIWSFC4ZZZIHBDHZAEQ, seg_pos={0 0}, upload=1507]: Done uploading piece. err: <nil> Jul 26 23:30:52.341522808 (1h39m8.122456179s) [noise=true , address=77.90.31.16:4753 , piece_id=OVD27UMNYNVQBMKRLGFCBVJAI55BCIZ5QOKZN4CYDZA3YB4MEDZA, seg_pos={0 0}, upload=1507]: Done uploading piece. err: <nil> Jul 26 23:30:53.099157711 (1h39m8.880091076s) [noise=true , address=104.36.82.4:4822 , piece_id=2WDNUC4O3T7XQCSNX75ITSZ5GX2ZX2OAYQ5F5PQZYRSOYGEMQQTQ, seg_pos={0 0}, upload=1507]: Done uploading piece. err: ecclient: upload failed (node:12fgLN9DGBGxWfBEePSsk84FTrnKoKWwNgoET3FpotfJxJ28f75, address:104.36.82.4:4822): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:30:53.099318447 (1h39m8.880251815s) [noise=true , address=104.36.82.42:4807 , piece_id=7OFW3XZRU3RAJM4Z2DOKXIQ45IFT74A2LMLZVI7HN5BZHCWZWF6Q, seg_pos={0 0}, upload=1507]: Done uploading piece. err: ecclient: upload failed (node:1iTiX3X69T9rKBxe1MLwpJEJPZyXVATQBP9cDEnQe2riXMGZeZ, address:104.36.82.42:4807): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:30:53.099377598 (1h39m8.880310977s) [noise=true , address=77.90.31.11:4577 , piece_id=JGVMAJUXDJNJC2F7EOSOQK7K7EZKSQEDO4DXVGY6QMZPXC6UW5KQ, seg_pos={0 0}, upload=1507]: Done uploading piece. err: ecclient: upload failed (node:12rKdimacehmW1Tp5Zp4DJFxafLhcEeNh61cfw8Bfefr9pRG2ei, address:77.90.31.11:4577): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:30:53.099446635 (1h39m8.880380013s) [noise=true , address=104.36.82.47:4525 , piece_id=AOKK77ACINTXERF4ULGUJESEHELMZEJYH3SNIJ4UUNACL6F6TPEA, seg_pos={0 0}, upload=1507]: Done uploading piece. err: ecclient: upload failed (node:1hRLixZtY6GptLU6MgGZAvnk4YvuDD6WbBFUePZSndN9UbpHhX, address:104.36.82.47:4525): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:30:53.099518396 (1h39m8.880451762s) [noise=true , address=104.36.82.54:4168 , piece_id=IQXVK6FZFAL7QUV4FDMQSP33FSHU36SVDPOUHEFDPIEA27AAERCA, seg_pos={0 0}, upload=1507]: Done uploading piece. err: ecclient: upload failed (node:1XH68Yk8yh2Qa8vcPRYAuLHdGR7Jy4KtSBKYKgwAJpdDvVu7up, address:104.36.82.54:4168): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:30:53.099562733 (1h39m8.880496095s) [noise=true , address=104.36.82.46:4485 , piece_id=GGP4SYW5Y2K5T6EIYZHVQOVHBWVNBE2X4VEKEYAKFNJZLCPWTZRA, seg_pos={0 0}, upload=1507]: Done uploading piece. err: ecclient: upload failed (node:12w74nT4D4F3tttRavcoKiPzyct4CxvGQfbn6EcVypHzaexCxNJ, address:104.36.82.46:4485): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:30:53.099594241 (1h39m8.880527627s) [seg_pos={0 0}, upload=1507]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:30:53.127154565 (1h39m8.908087933s) [upload=1507]: Done uploading segments... Jul 26 23:30:53.127185593 (1h39m8.908118971s) [upload=1507]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.030224056s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 30 out of 60 is failed (threshold=3) [........................X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.053267464s FAIL (2025-07-26 23:30:36 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.053301916s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 28 out of 60 is failed (threshold=3) [................................XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.035359682s FAIL (2025-07-26 23:30:14 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== l 26 23:29:44.083913233 (1h37m59.864846615s) [upload=1505]: Got next segment. Inline: false Jul 26 23:29:44.084148252 (1h37m59.865081638s) [upload=1505]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:29:44.084161734 (1h37m59.86509512s) [upload=1505]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:29:44.162780039 (1h37m59.943713421s) [seg_pos={0 0}, upload=1505]: Begin upload segment... Jul 26 23:29:44.162869192 (1h37m59.943802578s) [noise=true , address=45.91.116.6:4973 , piece_id=BNKRRZ3EKVVW2YZM3MA633GAGVR5JZS6D4UQ32ADXZQVK47OIU3A, seg_pos={0 0}, upload=1505]: Uploading piece... Jul 26 23:29:44.163152307 (1h37m59.944085687s) [seg_pos={0 0}, upload=1505]: Done begin upload segment. Jul 26 23:29:44.163177555 (1h37m59.944110941s) [noise=true , address=104.36.82.15:4259 , piece_id=PTR6YVWWULP5FOONF757O5WBLNOY6DAB4SBXS5KBONQDXFVAITXQ, seg_pos={0 0}, upload=1505]: Uploading piece... Jul 26 23:29:44.164073908 (1h37m59.945007294s) [noise=true , address=45.91.116.6:4974 , piece_id=PSWCHJBIWVEOECR6NXW2EAF2DBNAHEQAT342TJZYBZHOPRPPZ4ZA, seg_pos={0 0}, upload=1505]: Uploading piece... Jul 26 23:29:44.164435774 (1h37m59.945369159s) [noise=true , address=104.36.82.5:4837 , piece_id=AUCQLQSGFAFE7HB3XVND52T35EEF4WAWMH654JLGT5XNCKKPB6WQ, seg_pos={0 0}, upload=1505]: Uploading piece... Jul 26 23:29:44.164992099 (1h37m59.945925487s) [noise=true , address=104.36.82.11:4115 , piece_id=KA7YJCXACKAHCGTZV3BL66CVYO3USSJMLLLYWJBIFXXHOWK6AYJA, seg_pos={0 0}, upload=1505]: Uploading piece... Jul 26 23:29:44.165374414 (1h37m59.946307796s) [noise=true , address=77.90.31.14:4661 , piece_id=6FTESDCXAVSIDLAK54BATEGBD533YR7QO6PT4M55Y4BIJ5AK475A, seg_pos={0 0}, upload=1505]: Uploading piece... Jul 26 23:29:44.166050225 (1h37m59.946983611s) [noise=true , address=104.36.82.10:4078 , piece_id=MEEGSRQVTAADWLIPMGPDZDG7WTFYR25N2C45DQXHODQV7GNTEXWA, seg_pos={0 0}, upload=1505]: Uploading piece... Jul 26 23:29:44.166378140 (1h37m59.947311525s) [noise=true , address=104.36.82.3:4767 , piece_id=DEZ4VAK3URRWV3WWC4ZOBDAUKRS2WPA2AFSHZ5AXVUOIJKGG76YQ, seg_pos={0 0}, upload=1505]: Uploading piece... Jul 26 23:29:44.166827687 (1h37m59.947761074s) [noise=true , address=104.36.82.6:4874 , piece_id=GA3TBWJIQTJLK5LMPMMMO3A2Q7NVDZKWRL7IVSP6VEESPCBMUM4Q, seg_pos={0 0}, upload=1505]: Uploading piece... Jul 26 23:29:44.167187127 (1h37m59.948120511s) [noise=true , address=104.36.82.4:4804 , piece_id=UDG4OG32JXTTEZFPYEE43KPHGYGHJDZU3I2THI7WNVZYDENAZZHA, seg_pos={0 0}, upload=1505]: Uploading piece... Jul 26 23:29:44.167569944 (1h37m59.948503322s) [noise=true , address=45.91.116.4:4939 , piece_id=HE62HJOBQMPS5AP3NVCRMUJE2JPEALJYQ5QKBDIHPDEWUVAOQRIQ, seg_pos={0 0}, upload=1505]: Uploading piece... Jul 26 23:29:44.169323071 (1h37m59.950256452s) [noise=true , address=45.91.116.10:4083 , piece_id=UJKSBUTRW2RHXDMSUI5W4VB2G3O426KRW42YU77COFXWG2TY6CKA, seg_pos={0 0}, upload=1505]: Uploading piece... Jul 26 23:29:44.169842666 (1h37m59.950776045s) [noise=true , address=45.91.116.5:4298 , piece_id=DFZF4JCR4HAMVF2RPJO3CKBZOBBGYA6O2GYJRQWXZKU7CGC55NKQ, seg_pos={0 0}, upload=1505]: Uploading piece... Jul 26 23:29:44.170181559 (1h37m59.951114941s) [noise=true , address=77.90.31.12:4592 , piece_id=AXULTRZ6QO4UVAOXNX6ZYWS7VSJFIJNWQMMVOF6XSPEXGRPKK4QQ, seg_pos={0 0}, upload=1505]: Uploading piece... Jul 26 23:29:44.170519238 (1h37m59.951452616s) [noise=true , address=104.36.82.9:4007 , piece_id=WCYH3MQOK3GOA664FYSZFP75WX6QBR6D4WD5NRXOD63K235I2DEQ, seg_pos={0 0}, upload=1505]: Uploading piece... Jul 26 23:29:44.170845500 (1h37m59.951778882s) [noise=true , address=45.91.116.13:4154 , piece_id=LSVYKFVWAQVLYG3S53VRWKFA6LQLRLJNSOIWXNVQNSPW5MUSF5RA, seg_pos={0 0}, upload=1505]: Uploading piece... Jul 26 23:29:44.255803258 (1h38m0.036736642s) [upload=1505]: Next returned nil segment Jul 26 23:29:44.255826629 (1h38m0.036760019s) [upload=1505]: Waiting for error group managing segments... Jul 26 23:29:44.259850747 (1h38m0.040784125s) [noise=true , address=45.91.116.6:4974 , piece_id=PSWCHJBIWVEOECR6NXW2EAF2DBNAHEQAT342TJZYBZHOPRPPZ4ZA, seg_pos={0 0}, upload=1505]: Done uploading piece. err: <nil> Jul 26 23:29:44.259942575 (1h38m0.040875951s) [noise=true , address=45.91.116.6:4973 , piece_id=BNKRRZ3EKVVW2YZM3MA633GAGVR5JZS6D4UQ32ADXZQVK47OIU3A, seg_pos={0 0}, upload=1505]: Done uploading piece. err: <nil> Jul 26 23:29:44.260557764 (1h38m0.041491142s) [noise=true , address=45.91.116.4:4939 , piece_id=HE62HJOBQMPS5AP3NVCRMUJE2JPEALJYQ5QKBDIHPDEWUVAOQRIQ, seg_pos={0 0}, upload=1505]: Done uploading piece. err: <nil> Jul 26 23:29:44.260674599 (1h38m0.041607968s) [noise=true , address=45.91.116.10:4083 , piece_id=UJKSBUTRW2RHXDMSUI5W4VB2G3O426KRW42YU77COFXWG2TY6CKA, seg_pos={0 0}, upload=1505]: Done uploading piece. err: <nil> Jul 26 23:29:44.261133395 (1h38m0.042066756s) [noise=true , address=45.91.116.13:4154 , piece_id=LSVYKFVWAQVLYG3S53VRWKFA6LQLRLJNSOIWXNVQNSPW5MUSF5RA, seg_pos={0 0}, upload=1505]: Done uploading piece. err: <nil> Jul 26 23:29:44.262558087 (1h38m0.043491459s) [noise=true , address=77.90.31.14:4661 , piece_id=6FTESDCXAVSIDLAK54BATEGBD533YR7QO6PT4M55Y4BIJ5AK475A, seg_pos={0 0}, upload=1505]: Done uploading piece. err: <nil> Jul 26 23:29:44.603659318 (1h38m0.384592686s) [noise=true , address=45.91.116.5:4298 , piece_id=DFZF4JCR4HAMVF2RPJO3CKBZOBBGYA6O2GYJRQWXZKU7CGC55NKQ, seg_pos={0 0}, upload=1505]: Done uploading piece. err: <nil> Jul 26 23:30:14.054937617 (1h38m29.835870982s) [noise=true , address=104.36.82.11:4115 , piece_id=KA7YJCXACKAHCGTZV3BL66CVYO3USSJMLLLYWJBIFXXHOWK6AYJA, seg_pos={0 0}, upload=1505]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:41510->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:14.055052371 (1h38m29.835985735s) [noise=true , address=104.36.82.4:4804 , piece_id=UDG4OG32JXTTEZFPYEE43KPHGYGHJDZU3I2THI7WNVZYDENAZZHA, seg_pos={0 0}, upload=1505]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:14.055110822 (1h38m29.836044186s) [noise=true , address=104.36.82.9:4007 , piece_id=WCYH3MQOK3GOA664FYSZFP75WX6QBR6D4WD5NRXOD63K235I2DEQ, seg_pos={0 0}, upload=1505]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:14.055422839 (1h38m29.836356212s) [noise=true , address=104.36.82.10:4078 , piece_id=MEEGSRQVTAADWLIPMGPDZDG7WTFYR25N2C45DQXHODQV7GNTEXWA, seg_pos={0 0}, upload=1505]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:36222->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:14.055578459 (1h38m29.836511845s) [noise=true , address=104.36.82.3:4767 , piece_id=DEZ4VAK3URRWV3WWC4ZOBDAUKRS2WPA2AFSHZ5AXVUOIJKGG76YQ, seg_pos={0 0}, upload=1505]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:42064->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:14.055650669 (1h38m29.836584037s) [noise=true , address=77.90.31.12:4592 , piece_id=AXULTRZ6QO4UVAOXNX6ZYWS7VSJFIJNWQMMVOF6XSPEXGRPKK4QQ, seg_pos={0 0}, upload=1505]: Done uploading piece. err: ecclient: upload failed (node:12dBQJ4Q1FVnvhYDRb9JrgVFPyQew4z7jfWgSsF7qxif5Mm8ai3, address:77.90.31.12:4592): protocol: rpc: dial tcp 77.90.31.12:4592: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:14.055724967 (1h38m29.836658332s) [noise=true , address=104.36.82.15:4259 , piece_id=PTR6YVWWULP5FOONF757O5WBLNOY6DAB4SBXS5KBONQDXFVAITXQ, seg_pos={0 0}, upload=1505]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:14.055731152 (1h38m29.836664539s) [noise=true , address=104.36.82.6:4874 , piece_id=GA3TBWJIQTJLK5LMPMMMO3A2Q7NVDZKWRL7IVSP6VEESPCBMUM4Q, seg_pos={0 0}, upload=1505]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:49066->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:14.055820922 (1h38m29.836754301s) [noise=true , address=104.36.82.5:4837 , piece_id=AUCQLQSGFAFE7HB3XVND52T35EEF4WAWMH654JLGT5XNCKKPB6WQ, seg_pos={0 0}, upload=1505]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 159.223.143.39:36558->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:30:14.055889162 (1h38m29.836822545s) [seg_pos={0 0}, upload=1505]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:30:14.088690384 (1h38m29.869623767s) [upload=1505]: Done uploading segments... Jul 26 23:30:14.088721855 (1h38m29.869655239s) [upload=1505]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.035379037s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [..............................XXX.X....XX.X.X.X.......XXX...] duration: 921.384742ms FAIL (2025-07-26 23:30:11 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 921.829386ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [........X.X..........XX...X...X..XX.X...X..X.......X...X....] duration: 21.272553143s FAIL (2025-07-26 23:29:56 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 21.27260123s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [...............................XXX.X....XX.X.X.X.......XXX..] duration: 1.026905294s FAIL (2025-07-26 23:29:45 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.026961636s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.042175494s FAIL (2025-07-26 23:29:44 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0}, upload=1492]: Begin upload segment... Jul 26 23:28:44.177539400 (1h36m59.958472766s) [seg_pos={0 0}, upload=1492]: Done begin upload segment. Jul 26 23:28:44.177563886 (1h36m59.958497252s) [noise=true , address=79.127.220.101:4041 , piece_id=XNUUBPI2S6XEGQW2HPBLHSVZE2WCGMARD34WWISJ77QQNQ3JDPRA, seg_pos={0 0}, upload=1492]: Uploading piece... Jul 26 23:28:44.177584032 (1h36m59.958517417s) [noise=true , address=45.91.116.14:4206 , piece_id=ZPJGIY5HOKJM77KAX6ELYLOG7W7XCZ3EYGSSFN72M2CDOL5Q6RRA, seg_pos={0 0}, upload=1492]: Uploading piece... Jul 26 23:28:44.177967350 (1h36m59.958900734s) [noise=true , address=104.36.82.53:4156 , piece_id=2BHO5ITUKBL6HE4WTLBS3CNMDZLVTJA4PLXKQGEJSPZFPOX6TPJQ, seg_pos={0 0}, upload=1492]: Uploading piece... Jul 26 23:28:44.178169315 (1h36m59.959102702s) [noise=true , address=104.36.82.47:4535 , piece_id=B5SMQGGPFHGDPLB6AIGFT5LZ6OOWZOXO6F7YB5VSKEGMH7DHLV4Q, seg_pos={0 0}, upload=1492]: Uploading piece... Jul 26 23:28:44.178335462 (1h36m59.959268827s) [noise=true , address=79.127.205.245:4247 , piece_id=ZYGGZAARUTL5OBKP5VUHGNWKZUVSUKRJKNX4EMNODPVTUPYAS2AA, seg_pos={0 0}, upload=1492]: Uploading piece... Jul 26 23:28:44.178643920 (1h36m59.959577285s) [noise=true , address=79.127.205.235:4680 , piece_id=CGLCCXERTYTOIFZHD5WRPSPAD3YRGJUDBFY5Y5JURBHOKI7CASHQ, seg_pos={0 0}, upload=1492]: Uploading piece... Jul 26 23:28:44.180675769 (1h36m59.961609153s) [noise=true , address=79.127.219.38:4001 , piece_id=FS4UAJ7FT24HWFVW2RRCNOQHOVXMMPZEYLHV3BGWYT5BIW6IBSIA, seg_pos={0 0}, upload=1492]: Uploading piece... Jul 26 23:28:44.181247363 (1h36m59.962180749s) [noise=true , address=64.49.13.138:4483 , piece_id=WZA4ZLWOSF6UOD57FSBLRBO2GO634GII4DXOXSSTJWHUDSETCTJA, seg_pos={0 0}, upload=1492]: Uploading piece... Jul 26 23:28:44.182358336 (1h36m59.963291721s) [noise=true , address=79.127.205.246:4260 , piece_id=VO3AGX5BVLHUPTINHKID67TCQWFVLIQYH5D3BPUQRY56UEMECE3Q, seg_pos={0 0}, upload=1492]: Uploading piece... Jul 26 23:28:44.184781677 (1h36m59.965715062s) [noise=true , address=77.90.31.18:4840 , piece_id=ICDQGZAI4I3OSPV54FLD4SLPTZ4JVCJ3JF3S7OYJYNR6EYVXQ4DA, seg_pos={0 0}, upload=1492]: Uploading piece... Jul 26 23:28:44.185484961 (1h36m59.966418343s) [noise=true , address=104.36.82.54:4188 , piece_id=JCE3T4ENLWRXA7G3HD3J3WG7ETFE3V44HXMHC2M5UYFKFFBRS4AQ, seg_pos={0 0}, upload=1492]: Uploading piece... Jul 26 23:28:44.185880636 (1h36m59.966814022s) [noise=true , address=79.127.205.234:4603 , piece_id=T3SVB5ETQM5REKYC4IS5KSYXAFIE5C4ENO2IO6TUQHEXRBZVED4Q, seg_pos={0 0}, upload=1492]: Uploading piece... Jul 26 23:28:44.186258765 (1h36m59.967192148s) [noise=true , address=64.49.13.140:4514 , piece_id=6X2NLWQSRZEZXSYBZ4HLNUU7XMNKYRUXQ2SJES5BKC6BHGK4LTEA, seg_pos={0 0}, upload=1492]: Uploading piece... Jul 26 23:28:44.186977767 (1h36m59.967911143s) [noise=true , address=79.127.205.233:4624 , piece_id=3VQK6APCRGZJVZURKUXFI2HP3I3KAIAL5YP6TSJDNFGWWUKVA7RA, seg_pos={0 0}, upload=1492]: Uploading piece... Jul 26 23:28:44.187505632 (1h36m59.968439017s) [noise=true , address=79.127.201.215:4360 , piece_id=NKKPUAEB5CGB2AKUYPDH2QUEE2DD5XSSRN2D4UTKEIC646GLOY2Q, seg_pos={0 0}, upload=1492]: Uploading piece... Jul 26 23:28:44.187953163 (1h36m59.968886542s) [noise=true , address=104.36.82.46:4508 , piece_id=R4JCICLRRQSYA3PBDG6NL66KFSWBJCPZEPIMQZQ4GBRAZC25XHGA, seg_pos={0 0}, upload=1492]: Uploading piece... Jul 26 23:28:44.188411223 (1h36m59.96934461s) [noise=true , address=77.90.31.16:4736 , piece_id=AMQ7KZGGZU3NRLAUF2L6U6BRMGVJU3TM3LVDYBR2XVY7LV5IRZ5A, seg_pos={0 0}, upload=1492]: Uploading piece... Jul 26 23:28:44.188899333 (1h36m59.969832717s) [noise=true , address=104.36.82.56:4263 , piece_id=BZPH47IAEMPWTNMWO5HFWDMHO4XQT4AVLVU2MV36P6CWTRE2AWXA, seg_pos={0 0}, upload=1492]: Uploading piece... Jul 26 23:28:44.252425380 (1h37m0.033358743s) [upload=1492]: Next returned nil segment Jul 26 23:28:44.252446744 (1h37m0.03338014s) [upload=1492]: Waiting for error group managing segments... Jul 26 23:28:44.394707185 (1h37m0.175640562s) [noise=true , address=79.127.219.38:4001 , piece_id=FS4UAJ7FT24HWFVW2RRCNOQHOVXMMPZEYLHV3BGWYT5BIW6IBSIA, seg_pos={0 0}, upload=1492]: Done uploading piece. err: <nil> Jul 26 23:28:44.738138057 (1h37m0.519071425s) [noise=true , address=79.127.205.234:4603 , piece_id=T3SVB5ETQM5REKYC4IS5KSYXAFIE5C4ENO2IO6TUQHEXRBZVED4Q, seg_pos={0 0}, upload=1492]: Done uploading piece. err: <nil> Jul 26 23:28:44.758862795 (1h37m0.539796176s) [noise=true , address=79.127.205.246:4260 , piece_id=VO3AGX5BVLHUPTINHKID67TCQWFVLIQYH5D3BPUQRY56UEMECE3Q, seg_pos={0 0}, upload=1492]: Done uploading piece. err: <nil> Jul 26 23:28:44.804818504 (1h37m0.585751878s) [noise=true , address=79.127.205.233:4624 , piece_id=3VQK6APCRGZJVZURKUXFI2HP3I3KAIAL5YP6TSJDNFGWWUKVA7RA, seg_pos={0 0}, upload=1492]: Done uploading piece. err: <nil> Jul 26 23:28:44.816394293 (1h37m0.597327659s) [noise=true , address=79.127.205.245:4247 , piece_id=ZYGGZAARUTL5OBKP5VUHGNWKZUVSUKRJKNX4EMNODPVTUPYAS2AA, seg_pos={0 0}, upload=1492]: Done uploading piece. err: <nil> Jul 26 23:28:44.935132986 (1h37m0.716066354s) [noise=true , address=79.127.205.235:4680 , piece_id=CGLCCXERTYTOIFZHD5WRPSPAD3YRGJUDBFY5Y5JURBHOKI7CASHQ, seg_pos={0 0}, upload=1492]: Done uploading piece. err: <nil> Jul 26 23:28:44.984443218 (1h37m0.765376579s) [noise=true , address=64.49.13.138:4483 , piece_id=WZA4ZLWOSF6UOD57FSBLRBO2GO634GII4DXOXSSTJWHUDSETCTJA, seg_pos={0 0}, upload=1492]: Done uploading piece. err: <nil> Jul 26 23:28:46.176741845 (1h37m1.957675225s) [noise=true , address=45.91.116.14:4206 , piece_id=ZPJGIY5HOKJM77KAX6ELYLOG7W7XCZ3EYGSSFN72M2CDOL5Q6RRA, seg_pos={0 0}, upload=1492]: Done uploading piece. err: <nil> Jul 26 23:28:46.325510544 (1h37m2.106443925s) [noise=true , address=79.127.220.101:4041 , piece_id=XNUUBPI2S6XEGQW2HPBLHSVZE2WCGMARD34WWISJ77QQNQ3JDPRA, seg_pos={0 0}, upload=1492]: Done uploading piece. err: <nil> Jul 26 23:28:49.844214635 (1h37m5.625148s) [noise=true , address=64.49.13.140:4514 , piece_id=6X2NLWQSRZEZXSYBZ4HLNUU7XMNKYRUXQ2SJES5BKC6BHGK4LTEA, seg_pos={0 0}, upload=1492]: Done uploading piece. err: <nil> Jul 26 23:29:08.289364055 (1h37m24.070297421s) [noise=true , address=79.127.201.215:4360 , piece_id=NKKPUAEB5CGB2AKUYPDH2QUEE2DD5XSSRN2D4UTKEIC646GLOY2Q, seg_pos={0 0}, upload=1492]: Done uploading piece. err: <nil> Jul 26 23:29:44.061641040 (1h37m59.842574427s) [noise=true , address=104.36.82.46:4508 , piece_id=R4JCICLRRQSYA3PBDG6NL66KFSWBJCPZEPIMQZQ4GBRAZC25XHGA, seg_pos={0 0}, upload=1492]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: write tcp 159.223.143.39:55338->104.36.82.46:4508: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:44.061849251 (1h37m59.842782616s) [noise=true , address=104.36.82.53:4156 , piece_id=2BHO5ITUKBL6HE4WTLBS3CNMDZLVTJA4PLXKQGEJSPZFPOX6TPJQ, seg_pos={0 0}, upload=1492]: Done uploading piece. err: ecclient: upload failed (node:12cFTycUosmf2ynRZmHYT1axN7VcS31wjqv74CxUpb5nThvmW5, address:104.36.82.53:4156): protocol: write tcp 159.223.143.39:55228->104.36.82.53:4156: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:44.061912252 (1h37m59.842845821s) [noise=true , address=104.36.82.54:4188 , piece_id=JCE3T4ENLWRXA7G3HD3J3WG7ETFE3V44HXMHC2M5UYFKFFBRS4AQ, seg_pos={0 0}, upload=1492]: Done uploading piece. err: ecclient: upload failed (node:1HQ41iFa5jBBMhjL3LArqNfVhmKzNzF4N436n4qdod6GoNhAQG, address:104.36.82.54:4188): protocol: write tcp 159.223.143.39:39886->104.36.82.54:4188: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:44.061966593 (1h37m59.842899977s) [noise=true , address=77.90.31.16:4736 , piece_id=AMQ7KZGGZU3NRLAUF2L6U6BRMGVJU3TM3LVDYBR2XVY7LV5IRZ5A, seg_pos={0 0}, upload=1492]: Done uploading piece. err: ecclient: upload failed (node:1DqyGgLKeQEoD6e4TLmUJ6kQYPsZkKe9WNJf3HxWn3erAargYk, address:77.90.31.16:4736): protocol: write tcp 159.223.143.39:51728->77.90.31.16:4736: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:44.062047954 (1h37m59.842981337s) [noise=true , address=104.36.82.56:4263 , piece_id=BZPH47IAEMPWTNMWO5HFWDMHO4XQT4AVLVU2MV36P6CWTRE2AWXA, seg_pos={0 0}, upload=1492]: Done uploading piece. err: ecclient: upload failed (node:12tLW4r8AQKLCbSWZoNUVi5v2YtF8NthUmX81TLQn8G8TotuByG, address:104.36.82.56:4263): protocol: write tcp 159.223.143.39:33148->104.36.82.56:4263: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:44.062109242 (1h37m59.843042619s) [noise=true , address=77.90.31.18:4840 , piece_id=ICDQGZAI4I3OSPV54FLD4SLPTZ4JVCJ3JF3S7OYJYNR6EYVXQ4DA, seg_pos={0 0}, upload=1492]: Done uploading piece. err: ecclient: upload failed (node:12HcxQABJsUpjxiYdAkRFTpRTrT4KfrP8RKZkUhsyea1EKmwd5V, address:77.90.31.18:4840): protocol: read tcp 159.223.143.39:59488->77.90.31.18:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:44.062254361 (1h37m59.843187743s) [noise=true , address=104.36.82.47:4535 , piece_id=B5SMQGGPFHGDPLB6AIGFT5LZ6OOWZOXO6F7YB5VSKEGMH7DHLV4Q, seg_pos={0 0}, upload=1492]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:29:44.062273525 (1h37m59.843206913s) [seg_pos={0 0}, upload=1492]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:29:44.101766359 (1h37m59.882699729s) [upload=1492]: Done uploading segments... Jul 26 23:29:44.101793455 (1h37m59.882726831s) [upload=1492]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.042194316s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 27 out of 60 is failed (threshold=3) [.................................XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.047108846s FAIL (2025-07-26 23:29:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 62170 (1h37m29.766495557s) [upload=1498]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:29:13.985587188 (1h37m29.766520567s) [upload=1498]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:29:14.016200656 (1h37m29.797134042s) [seg_pos={0 0}, upload=1498]: Begin upload segment... Jul 26 23:29:14.016271302 (1h37m29.797204685s) [seg_pos={0 0}, upload=1498]: Done begin upload segment. Jul 26 23:29:14.016301769 (1h37m29.797235156s) [noise=true , address=104.36.82.10:4077 , piece_id=25V5IQYQUKOBF74SXGTZFSXYZ3QCPUXQG2VOXU5VG2AGSQ7S44LA, seg_pos={0 0}, upload=1498]: Uploading piece... Jul 26 23:29:14.016980150 (1h37m29.797913538s) [noise=true , address=104.36.82.4:4802 , piece_id=PLDIX7IYC6FM3X5KVBJZDSUNISYRNN3MFMLDKGBD3MYC6VPPLI4Q, seg_pos={0 0}, upload=1498]: Uploading piece... Jul 26 23:29:14.017463399 (1h37m29.798396784s) [noise=true , address=104.36.82.8:4972 , piece_id=M53ZGYDQE526XDRHC5H7ACHNEGR35IV3OE7Y235TWU5E6WIM2OYQ, seg_pos={0 0}, upload=1498]: Uploading piece... Jul 26 23:29:14.017964521 (1h37m29.798897907s) [noise=true , address=45.91.116.8:4048 , piece_id=RHZ442LMVDKIX6H4DRYX5SPUCNONI2MVMSQHOWVK7LKTLB57NOEQ, seg_pos={0 0}, upload=1498]: Uploading piece... Jul 26 23:29:14.018472123 (1h37m29.799405512s) [noise=true , address=104.36.82.9:4007 , piece_id=UPNS4KCEL77PWHEFGJP3SVJ7LTHRZGPVLB3ITAYRDUBZVIU7QYRA, seg_pos={0 0}, upload=1498]: Uploading piece... Jul 26 23:29:14.018955110 (1h37m29.799888495s) [noise=true , address=104.36.82.13:4187 , piece_id=6WSBO4M7QMNQMEXKEM7ZN75PZ6NOPDRPTDM7XUFQZVRQAP7PXJ5Q, seg_pos={0 0}, upload=1498]: Uploading piece... Jul 26 23:29:14.019463806 (1h37m29.800397191s) [noise=true , address=77.90.31.21:4879 , piece_id=OGQKXLJG65YY25XMCCKX2OBF5UEVERNONXGL3GQED7XP6NKG5XXQ, seg_pos={0 0}, upload=1498]: Uploading piece... Jul 26 23:29:14.019867461 (1h37m29.800800845s) [noise=true , address=45.91.116.7:4009 , piece_id=AZO36VEATGGIHUTWA3GNIZEV7SL22WCGNBZ6VVSSREXWFMTVZIBQ, seg_pos={0 0}, upload=1498]: Uploading piece... Jul 26 23:29:14.020433639 (1h37m29.801367025s) [noise=true , address=77.90.31.14:4662 , piece_id=BKTLRUHWQDJURVLX4VU2GTUDYBOCHW45TDBMGYZWO7FKFOYDLMHA, seg_pos={0 0}, upload=1498]: Uploading piece... Jul 26 23:29:14.020848040 (1h37m29.801781425s) [noise=true , address=45.91.116.10:4082 , piece_id=MZWXDPRBPT6IRCZPLDAWXUIYTSDNGO2ZES7ALVLT7XNQTTAJHSYA, seg_pos={0 0}, upload=1498]: Uploading piece... Jul 26 23:29:14.021226962 (1h37m29.802160346s) [noise=true , address=104.36.82.6:4876 , piece_id=HFXQAW2QMP76MSYSPBFBN5R6UFGSYZA4XVOSWYEOK5S4E7NSKZ5A, seg_pos={0 0}, upload=1498]: Uploading piece... Jul 26 23:29:14.021572347 (1h37m29.802505735s) [noise=true , address=104.36.82.11:4114 , piece_id=LVMKVX5XZ5DKHURD2CGEBU3D2JI2X7VLEG3HGPMIDBPD2CQPOWCQ, seg_pos={0 0}, upload=1498]: Uploading piece... Jul 26 23:29:14.021932595 (1h37m29.802865981s) [noise=true , address=45.91.116.5:4298 , piece_id=GDW37X4U3HPPA3RYIHQXA3KLK467GYHSDEMWG6EUU5L2OYSQMCAQ, seg_pos={0 0}, upload=1498]: Uploading piece... Jul 26 23:29:14.022288139 (1h37m29.803221525s) [noise=true , address=104.36.82.15:4259 , piece_id=6T6UUUWCAEXENC62E2AOBFXGZGEUS3IF33EK6YTFW2JB62NDNNNA, seg_pos={0 0}, upload=1498]: Uploading piece... Jul 26 23:29:14.022607022 (1h37m29.803540407s) [noise=true , address=104.36.82.6:4874 , piece_id=4H5HVEEQQQDPOS53CGRIHTZOUIZKM7FAGNE2S3LUE2YA7R6WOBAA, seg_pos={0 0}, upload=1498]: Uploading piece... Jul 26 23:29:14.022965618 (1h37m29.803899002s) [noise=true , address=104.36.82.2:4729 , piece_id=6Z7JMTFT27SJONHTTM24KS7Y2Y3IGUHVH5TUHNL65GS2AXCY2ABA, seg_pos={0 0}, upload=1498]: Uploading piece... Jul 26 23:29:14.109372583 (1h37m29.890305949s) [upload=1498]: Next returned nil segment Jul 26 23:29:14.109395561 (1h37m29.890328948s) [upload=1498]: Waiting for error group managing segments... Jul 26 23:29:14.113485386 (1h37m29.894418756s) [noise=true , address=45.91.116.10:4082 , piece_id=MZWXDPRBPT6IRCZPLDAWXUIYTSDNGO2ZES7ALVLT7XNQTTAJHSYA, seg_pos={0 0}, upload=1498]: Done uploading piece. err: <nil> Jul 26 23:29:14.113631499 (1h37m29.894564865s) [noise=true , address=45.91.116.5:4298 , piece_id=GDW37X4U3HPPA3RYIHQXA3KLK467GYHSDEMWG6EUU5L2OYSQMCAQ, seg_pos={0 0}, upload=1498]: Done uploading piece. err: <nil> Jul 26 23:29:14.113956270 (1h37m29.89488964s) [noise=true , address=45.91.116.7:4009 , piece_id=AZO36VEATGGIHUTWA3GNIZEV7SL22WCGNBZ6VVSSREXWFMTVZIBQ, seg_pos={0 0}, upload=1498]: Done uploading piece. err: <nil> Jul 26 23:29:14.114168789 (1h37m29.895102158s) [noise=true , address=45.91.116.8:4048 , piece_id=RHZ442LMVDKIX6H4DRYX5SPUCNONI2MVMSQHOWVK7LKTLB57NOEQ, seg_pos={0 0}, upload=1498]: Done uploading piece. err: <nil> Jul 26 23:29:14.114994775 (1h37m29.89592816s) [noise=true , address=77.90.31.14:4662 , piece_id=BKTLRUHWQDJURVLX4VU2GTUDYBOCHW45TDBMGYZWO7FKFOYDLMHA, seg_pos={0 0}, upload=1498]: Done uploading piece. err: <nil> Jul 26 23:29:14.117163399 (1h37m29.898096766s) [noise=true , address=77.90.31.21:4879 , piece_id=OGQKXLJG65YY25XMCCKX2OBF5UEVERNONXGL3GQED7XP6NKG5XXQ, seg_pos={0 0}, upload=1498]: Done uploading piece. err: <nil> Jul 26 23:29:43.954252562 (1h37m59.735185927s) [noise=true , address=104.36.82.4:4802 , piece_id=PLDIX7IYC6FM3X5KVBJZDSUNISYRNN3MFMLDKGBD3MYC6VPPLI4Q, seg_pos={0 0}, upload=1498]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: rpc: dial tcp 104.36.82.4:4802: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:43.954314090 (1h37m59.735247469s) [noise=true , address=104.36.82.8:4972 , piece_id=M53ZGYDQE526XDRHC5H7ACHNEGR35IV3OE7Y235TWU5E6WIM2OYQ, seg_pos={0 0}, upload=1498]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:43.954404440 (1h37m59.735337819s) [noise=true , address=104.36.82.6:4876 , piece_id=HFXQAW2QMP76MSYSPBFBN5R6UFGSYZA4XVOSWYEOK5S4E7NSKZ5A, seg_pos={0 0}, upload=1498]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: rpc: dial tcp 104.36.82.6:4876: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:43.954490869 (1h37m59.735424249s) [noise=true , address=104.36.82.15:4259 , piece_id=6T6UUUWCAEXENC62E2AOBFXGZGEUS3IF33EK6YTFW2JB62NDNNNA, seg_pos={0 0}, upload=1498]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:43.954508049 (1h37m59.735441413s) [noise=true , address=104.36.82.2:4729 , piece_id=6Z7JMTFT27SJONHTTM24KS7Y2Y3IGUHVH5TUHNL65GS2AXCY2ABA, seg_pos={0 0}, upload=1498]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:43.954580692 (1h37m59.735514068s) [noise=true , address=104.36.82.10:4077 , piece_id=25V5IQYQUKOBF74SXGTZFSXYZ3QCPUXQG2VOXU5VG2AGSQ7S44LA, seg_pos={0 0}, upload=1498]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:43.954596270 (1h37m59.735529635s) [noise=true , address=104.36.82.9:4007 , piece_id=UPNS4KCEL77PWHEFGJP3SVJ7LTHRZGPVLB3ITAYRDUBZVIU7QYRA, seg_pos={0 0}, upload=1498]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:43.954653160 (1h37m59.735586539s) [noise=true , address=104.36.82.13:4187 , piece_id=6WSBO4M7QMNQMEXKEM7ZN75PZ6NOPDRPTDM7XUFQZVRQAP7PXJ5Q, seg_pos={0 0}, upload=1498]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 159.223.143.39:48806->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:43.954658719 (1h37m59.735592083s) [noise=true , address=104.36.82.11:4114 , piece_id=LVMKVX5XZ5DKHURD2CGEBU3D2JI2X7VLEG3HGPMIDBPD2CQPOWCQ, seg_pos={0 0}, upload=1498]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: write tcp 159.223.143.39:59572->104.36.82.11:4114: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:43.954664084 (1h37m59.735597453s) [noise=true , address=104.36.82.6:4874 , piece_id=4H5HVEEQQQDPOS53CGRIHTZOUIZKM7FAGNE2S3LUE2YA7R6WOBAA, seg_pos={0 0}, upload=1498]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:29:43.954705429 (1h37m59.735638814s) [seg_pos={0 0}, upload=1498]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:29:43.998606753 (1h37m59.779540125s) [upload=1498]: Done uploading segments... Jul 26 23:29:43.998631051 (1h37m59.77956443s) [upload=1498]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.04713877s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 30 out of 60 is failed (threshold=3) [X........................X......XXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.030894437s FAIL (2025-07-26 23:29:41 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.030924723s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [................................XXX.X....XX.X.X.X.......XXX.] duration: 23.506336422s FAIL (2025-07-26 23:29:12 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 23.506363029s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.........X.X..........XX...X...X..XX.X...X..X.......X...X...] duration: 40.112408827s FAIL (2025-07-26 23:29:07 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.112449374s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.033449578s FAIL (2025-07-26 23:28:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 476]: Uploading piece... Jul 26 23:27:44.021758647 (1h35m59.802692031s) [noise=true , address=79.127.205.233:4650 , piece_id=OETWVLOPJ4NJYPSWZOFN6NLIZK5HRJRH4E66MESIDFH75ZOWHOKA, seg_pos={0 0}, upload=1476]: Uploading piece... Jul 26 23:27:44.022020380 (1h35m59.802953757s) [noise=true , address=77.90.31.18:4838 , piece_id=6KEDOVA3YL5MZQ352A77JSM2UZRYUDI5TALJC6Z7XP3XOJ7WUF3Q, seg_pos={0 0}, upload=1476]: Uploading piece... Jul 26 23:27:44.022094519 (1h35m59.803027919s) [noise=true , address=104.36.82.53:4145 , piece_id=6SNESZCCLG4OIZVXFY5JPUSE4ATRX7VOTXDE2BQA7IKL47MW3HJA, seg_pos={0 0}, upload=1476]: Uploading piece... Jul 26 23:27:44.022466654 (1h35m59.80340004s) [noise=true , address=79.127.205.234:4606 , piece_id=WILNJXK2FOZJSBSOH6ERXRQH4ANC2N2LEKBVFA6AVZKCYCQWZWVQ, seg_pos={0 0}, upload=1476]: Uploading piece... Jul 26 23:27:44.022998822 (1h35m59.803932206s) [noise=true , address=64.49.13.154:4927 , piece_id=HXZANNOWAGH26I5RDDPDI3ATEOQXQLJ4N23GM6GCWDQZ27WPM6HQ, seg_pos={0 0}, upload=1476]: Uploading piece... Jul 26 23:27:44.023420153 (1h35m59.804353528s) [noise=true , address=104.36.82.52:4113 , piece_id=IAF2W52XFV73WEOQYGA5XFBFVLU2GXPY5LJOUUFDQEDIRVHWEKJQ, seg_pos={0 0}, upload=1476]: Uploading piece... Jul 26 23:27:44.023773456 (1h35m59.804706832s) [noise=true , address=104.36.82.54:4178 , piece_id=MBXUJ5LVOPLDIPGTGCQZKNE3LWD6JZPKNWUTLONCEZE3QYBH7FXA, seg_pos={0 0}, upload=1476]: Uploading piece... Jul 26 23:27:44.024413568 (1h35m59.805346956s) [noise=true , address=104.36.82.46:4505 , piece_id=X5FDBIU37NWVFVGTJBIRVRUD3RYZT44YWICGDEAJIOXOXSAVKSLQ, seg_pos={0 0}, upload=1476]: Uploading piece... Jul 26 23:27:44.024770171 (1h35m59.805703555s) [noise=true , address=104.36.82.42:4829 , piece_id=YEIXWGTYQWQXW5Q5LS2ETCGQ6XNH56HZPLTA2R7JJEWFRJMGLT4A, seg_pos={0 0}, upload=1476]: Uploading piece... Jul 26 23:27:44.025133227 (1h35m59.806066606s) [noise=true , address=77.90.31.21:4912 , piece_id=356CAHRDMQP74XEFU6U6NSJZB6SHPJAIBNIXBHWSW26ABZTDTQNQ, seg_pos={0 0}, upload=1476]: Uploading piece... Jul 26 23:27:44.025497052 (1h35m59.806430439s) [noise=true , address=79.127.201.215:4345 , piece_id=NP6MSGOW66DO56PP4GO4DOK7LTHJ76QVECWIPRN7T6D35CFTP65A, seg_pos={0 0}, upload=1476]: Uploading piece... Jul 26 23:27:44.025895581 (1h35m59.806828964s) [noise=true , address=79.127.220.101:4041 , piece_id=IQMVXJ6HWUTZOIXPXX2CNILXKV6JPXHK52GJXATJVWFFLYAKLNCQ, seg_pos={0 0}, upload=1476]: Uploading piece... Jul 26 23:27:44.026362396 (1h35m59.807295783s) [noise=true , address=104.36.82.47:4533 , piece_id=RSPQFWXHVSG2LA7PIC626OWP7X67XGGWXARJWNHMJLDEFAQ5RUEQ, seg_pos={0 0}, upload=1476]: Uploading piece... Jul 26 23:27:44.026817138 (1h35m59.807750516s) [noise=true , address=104.36.82.56:4235 , piece_id=QAJQ4KWD5SSHM2ZA4MWF2LSYW7NXSHLD3WGXCOJTQFAFLQU6OT7Q, seg_pos={0 0}, upload=1476]: Uploading piece... Jul 26 23:27:44.093483716 (1h35m59.874417102s) [upload=1476]: Next returned nil segment Jul 26 23:27:44.093505383 (1h35m59.874438765s) [upload=1476]: Waiting for error group managing segments... Jul 26 23:27:44.136754042 (1h35m59.917687408s) [noise=true , address=77.90.31.21:4912 , piece_id=356CAHRDMQP74XEFU6U6NSJZB6SHPJAIBNIXBHWSW26ABZTDTQNQ, seg_pos={0 0}, upload=1476]: Done uploading piece. err: <nil> Jul 26 23:27:44.622476232 (1h36m0.403409598s) [noise=true , address=79.127.205.233:4650 , piece_id=OETWVLOPJ4NJYPSWZOFN6NLIZK5HRJRH4E66MESIDFH75ZOWHOKA, seg_pos={0 0}, upload=1476]: Done uploading piece. err: <nil> Jul 26 23:27:44.629953164 (1h36m0.41088654s) [noise=true , address=79.127.205.234:4606 , piece_id=WILNJXK2FOZJSBSOH6ERXRQH4ANC2N2LEKBVFA6AVZKCYCQWZWVQ, seg_pos={0 0}, upload=1476]: Done uploading piece. err: <nil> Jul 26 23:27:44.756417044 (1h36m0.537350407s) [noise=true , address=64.49.13.138:4482 , piece_id=PF35PFPNS23MCE7QWFZSPAMBORH5EIP2F3S56X76U2EQLVWTBO7A, seg_pos={0 0}, upload=1476]: Done uploading piece. err: <nil> Jul 26 23:27:45.424482459 (1h36m1.20541582s) [noise=true , address=79.127.220.101:4041 , piece_id=IQMVXJ6HWUTZOIXPXX2CNILXKV6JPXHK52GJXATJVWFFLYAKLNCQ, seg_pos={0 0}, upload=1476]: Done uploading piece. err: <nil> Jul 26 23:27:53.699246467 (1h36m9.480179836s) [noise=true , address=64.49.13.154:4927 , piece_id=HXZANNOWAGH26I5RDDPDI3ATEOQXQLJ4N23GM6GCWDQZ27WPM6HQ, seg_pos={0 0}, upload=1476]: Done uploading piece. err: <nil> Jul 26 23:28:21.830787781 (1h36m37.611721159s) [noise=true , address=79.127.201.215:4345 , piece_id=NP6MSGOW66DO56PP4GO4DOK7LTHJ76QVECWIPRN7T6D35CFTP65A, seg_pos={0 0}, upload=1476]: Done uploading piece. err: <nil> Jul 26 23:28:43.948346852 (1h36m59.729280233s) [noise=true , address=77.90.31.18:4838 , piece_id=6KEDOVA3YL5MZQ352A77JSM2UZRYUDI5TALJC6Z7XP3XOJ7WUF3Q, seg_pos={0 0}, upload=1476]: Done uploading piece. err: ecclient: upload failed (node:1if7vf1cNTBwAhb6owi6tuGLYqSfa3feWhsvGSevwuwykhz2RY, address:77.90.31.18:4838): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:28:43.948625207 (1h36m59.729558582s) [noise=true , address=104.36.82.55:4210 , piece_id=NQ4FTWHZ5O5LSAILRSV3Q23RIWWNOGYDNWWXCORY3OVKHXKFFDGA, seg_pos={0 0}, upload=1476]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: rpc: dial tcp 104.36.82.55:4210: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:43.948781634 (1h36m59.72971502s) [noise=true , address=104.36.82.42:4829 , piece_id=YEIXWGTYQWQXW5Q5LS2ETCGQ6XNH56HZPLTA2R7JJEWFRJMGLT4A, seg_pos={0 0}, upload=1476]: Done uploading piece. err: ecclient: upload failed (node:1F5T8Vt1x4nXFaSenvGBCYjxhRL1H1sxRvzLjW3CqzwDVhq39F, address:104.36.82.42:4829): protocol: write tcp 159.223.143.39:34254->104.36.82.42:4829: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:43.948856934 (1h36m59.72979031s) [noise=true , address=104.36.82.47:4533 , piece_id=RSPQFWXHVSG2LA7PIC626OWP7X67XGGWXARJWNHMJLDEFAQ5RUEQ, seg_pos={0 0}, upload=1476]: Done uploading piece. err: ecclient: upload failed (node:1JUUdAnaUPAE8tkzPvVCokFbZMwmhqZvwKHXReK9VU2QeDqavR, address:104.36.82.47:4533): protocol: rpc: dial tcp 104.36.82.47:4533: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:43.948914008 (1h36m59.729847388s) [noise=true , address=104.36.82.53:4145 , piece_id=6SNESZCCLG4OIZVXFY5JPUSE4ATRX7VOTXDE2BQA7IKL47MW3HJA, seg_pos={0 0}, upload=1476]: Done uploading piece. err: ecclient: upload failed (node:1rXBNYi6SgZdwGbiU5ooBE8U7DrEoAGw25ncYsRETgNh4CsiLX, address:104.36.82.53:4145): protocol: write tcp 159.223.143.39:60544->104.36.82.53:4145: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:43.948931878 (1h36m59.729865256s) [noise=true , address=104.36.82.46:4505 , piece_id=X5FDBIU37NWVFVGTJBIRVRUD3RYZT44YWICGDEAJIOXOXSAVKSLQ, seg_pos={0 0}, upload=1476]: Done uploading piece. err: ecclient: upload failed (node:1i6itGoBhLsLiqC8et8Y5L62fLjcWxy18u39NAr4R3nXNMWcsf, address:104.36.82.46:4505): protocol: write tcp 159.223.143.39:48258->104.36.82.46:4505: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:43.948986378 (1h36m59.729919755s) [noise=true , address=77.90.31.15:4706 , piece_id=ZJHHWAEWAOPBNY7YLNURAW2ZAVFMVAI2QIU6GYL2QDDRYQH5XTRA, seg_pos={0 0}, upload=1476]: Done uploading piece. err: ecclient: upload failed (node:12YZZnCo54XknuvoBoVtRudXSnoWdNdrn23Lp6Gou3g451BzUwG, address:77.90.31.15:4706): protocol: read tcp 159.223.143.39:51542->77.90.31.15:4706: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:43.949001281 (1h36m59.729934649s) [noise=true , address=104.36.82.56:4235 , piece_id=QAJQ4KWD5SSHM2ZA4MWF2LSYW7NXSHLD3WGXCOJTQFAFLQU6OT7Q, seg_pos={0 0}, upload=1476]: Done uploading piece. err: ecclient: upload failed (node:1a1sW1G1WPL5WQ2XxprWKQMsJkcb5ETSkk3ULp1TjRaZWptaT, address:104.36.82.56:4235): protocol: write tcp 159.223.143.39:40980->104.36.82.56:4235: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:43.949037838 (1h36m59.729971214s) [noise=true , address=104.36.82.13:4198 , piece_id=IDAAH2GJZXZIVXN3FBU2VAAWXARNK5C3J2LWKLN2R2YCESWSTRQQ, seg_pos={0 0}, upload=1476]: Done uploading piece. err: ecclient: upload failed (node:1Fj2mNZexPwM4B48kxLin8GF17KJN1RxNcZjGkD36LDviUevFm, address:104.36.82.13:4198): protocol: rpc: dial tcp 104.36.82.13:4198: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:43.949101083 (1h36m59.730034461s) [noise=true , address=104.36.82.52:4113 , piece_id=IAF2W52XFV73WEOQYGA5XFBFVLU2GXPY5LJOUUFDQEDIRVHWEKJQ, seg_pos={0 0}, upload=1476]: Done uploading piece. err: ecclient: upload failed (node:1brZ4RtcK5J3xCPmnbsTccF9hrnj8tqhV1WyMqqrjuitgSGMou, address:104.36.82.52:4113): protocol: write tcp 159.223.143.39:33342->104.36.82.52:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:43.949109506 (1h36m59.730042882s) [noise=true , address=104.36.82.54:4178 , piece_id=MBXUJ5LVOPLDIPGTGCQZKNE3LWD6JZPKNWUTLONCEZE3QYBH7FXA, seg_pos={0 0}, upload=1476]: Done uploading piece. err: ecclient: upload failed (node:1jyRCtxSCu2bkU41Kgb1aeGppQLgX5iEo322Zoi1cWFkmGeYgr, address:104.36.82.54:4178): protocol: write tcp 159.223.143.39:47388->104.36.82.54:4178: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:43.949128280 (1h36m59.730061643s) [seg_pos={0 0}, upload=1476]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 23:28:43.979306063 (1h36m59.760239432s) [upload=1476]: Done uploading segments... Jul 26 23:28:43.979329240 (1h36m59.760262619s) [upload=1476]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.033478857s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 26 out of 60 is failed (threshold=3) [..................................XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.033890826s FAIL (2025-07-26 23:28:40 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== 2]: Done starting upload Jul 26 23:28:10.141071192 (1h36m25.922004575s) [upload=1482]: Uploading segments... Jul 26 23:28:10.141581331 (1h36m25.922514709s) [upload=1482]: Got next segment. Inline: false Jul 26 23:28:10.141659568 (1h36m25.922592934s) [upload=1482]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:28:10.141667906 (1h36m25.922601284s) [upload=1482]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:28:10.171810525 (1h36m25.952743924s) [seg_pos={0 0}, upload=1482]: Begin upload segment... Jul 26 23:28:10.171863851 (1h36m25.952797213s) [seg_pos={0 0}, upload=1482]: Done begin upload segment. Jul 26 23:28:10.171891329 (1h36m25.952824717s) [noise=true , address=104.36.82.15:4259 , piece_id=U7VC6KV5S6OMUMWH3WCG6B463576WKW56Q7PC7GMMF3BMPNRO4WA, seg_pos={0 0}, upload=1482]: Uploading piece... Jul 26 23:28:10.172469632 (1h36m25.953403016s) [noise=true , address=45.91.116.12:4227 , piece_id=42RNXBMHTZPW5NDK2SPBEBGTQUNCQIJC33YQJWCEPBBXY5OWJRYQ, seg_pos={0 0}, upload=1482]: Uploading piece... Jul 26 23:28:10.172873534 (1h36m25.953806922s) [noise=true , address=104.36.82.16:4295 , piece_id=B26KO53SRNRXP4M4QTOR6M27BTXQH6LBCWVWQDVJFTV24NNIXDAA, seg_pos={0 0}, upload=1482]: Uploading piece... Jul 26 23:28:10.173240846 (1h36m25.95417423s) [noise=true , address=45.91.116.5:4298 , piece_id=2TR6DDGKF56VSWNV76UG6JMVKJUDJ6U3TMD5YHRWJB3MPLYQ3KEQ, seg_pos={0 0}, upload=1482]: Uploading piece... Jul 26 23:28:10.173590609 (1h36m25.954523994s) [noise=true , address=104.36.82.6:4874 , piece_id=63GE366FL34LZEGSMM2ZDR2KXNMNQUVTZSSNSAYKY6YG5CI6NCSQ, seg_pos={0 0}, upload=1482]: Uploading piece... Jul 26 23:28:10.173945837 (1h36m25.954879226s) [noise=true , address=104.36.82.13:4187 , piece_id=IMYJAMS4O5AAZBPUPW2OXFT642D2ASXKWNIY4FGNDBNE7XH7NYTQ, seg_pos={0 0}, upload=1482]: Uploading piece... Jul 26 23:28:10.174310526 (1h36m25.955243911s) [noise=true , address=45.91.116.4:4938 , piece_id=UA6WPHN75IVG7X4YZRYNOEJVXBNPCEEWFVWWGWQ2GC6SPNHE6JBQ, seg_pos={0 0}, upload=1482]: Uploading piece... Jul 26 23:28:10.174666331 (1h36m25.955599724s) [noise=true , address=45.91.116.14:4191 , piece_id=34ZCJ32D6IJFYF2AQ4C4GCVKPZ3U3L5TITHRU57XDYNONEH223IA, seg_pos={0 0}, upload=1482]: Uploading piece... Jul 26 23:28:10.175112828 (1h36m25.956046216s) [noise=true , address=104.36.82.8:4971 , piece_id=YQNSSA6XJ7TRSSFSW3NJV6CGNZDOGU2DWSDHKMDQQB7IVKYJUSUA, seg_pos={0 0}, upload=1482]: Uploading piece... Jul 26 23:28:10.175619788 (1h36m25.956553172s) [noise=true , address=104.36.82.3:4765 , piece_id=4ZATPHYSOFLNXRDIWYDLDXBJCT2NUWFKAT33NO6ENZM3U2TNG4FQ, seg_pos={0 0}, upload=1482]: Uploading piece... Jul 26 23:28:10.176128803 (1h36m25.957062204s) [noise=true , address=45.91.116.8:4045 , piece_id=M4G2E74CPYOAQ3URGLNM2YAEJENXLYV36ETMJFC5J2AWRLNGSW7A, seg_pos={0 0}, upload=1482]: Uploading piece... Jul 26 23:28:10.176543599 (1h36m25.957476975s) [noise=true , address=104.36.82.7:4912 , piece_id=Y6GI4JJSUWSYJABJXXFUJR6WHHSLJRZ34TV2DQT52XV3TZ3CTW3A, seg_pos={0 0}, upload=1482]: Uploading piece... Jul 26 23:28:10.176832848 (1h36m25.957766245s) [noise=true , address=45.91.116.6:4973 , piece_id=ES2AH3CA4DB72HPL3GZOOCSL254GK4LX6FS5UMRJKXXU6U3TVVHA, seg_pos={0 0}, upload=1482]: Uploading piece... Jul 26 23:28:10.177329596 (1h36m25.958262987s) [noise=true , address=45.91.116.10:4082 , piece_id=TMQSMMFO75LXGWZUWFDBD6EXT5BISHUSKQTM74ZLDZBLYJNTXDEQ, seg_pos={0 0}, upload=1482]: Uploading piece... Jul 26 23:28:10.177754840 (1h36m25.958688225s) [noise=true , address=104.36.82.9:4007 , piece_id=BUX7CRWB3WSYLBARFJAPYAZG46532J26OCPMMBW6PFCRUY4YI4YA, seg_pos={0 0}, upload=1482]: Uploading piece... Jul 26 23:28:10.178063469 (1h36m25.958996894s) [noise=true , address=104.36.82.5:4838 , piece_id=6O7SVTD7JYHBFBRZYZNWBUDZ64GXMHLDPLB7VOCKJMBR3T2ZVYHA, seg_pos={0 0}, upload=1482]: Uploading piece... Jul 26 23:28:10.280056670 (1h36m26.060990047s) [upload=1482]: Next returned nil segment Jul 26 23:28:10.280079614 (1h36m26.061012994s) [upload=1482]: Waiting for error group managing segments... Jul 26 23:28:10.283878586 (1h36m26.064811962s) [noise=true , address=45.91.116.4:4938 , piece_id=UA6WPHN75IVG7X4YZRYNOEJVXBNPCEEWFVWWGWQ2GC6SPNHE6JBQ, seg_pos={0 0}, upload=1482]: Done uploading piece. err: <nil> Jul 26 23:28:10.284712429 (1h36m26.065645794s) [noise=true , address=45.91.116.8:4045 , piece_id=M4G2E74CPYOAQ3URGLNM2YAEJENXLYV36ETMJFC5J2AWRLNGSW7A, seg_pos={0 0}, upload=1482]: Done uploading piece. err: <nil> Jul 26 23:28:10.285259537 (1h36m26.066192907s) [noise=true , address=45.91.116.10:4082 , piece_id=TMQSMMFO75LXGWZUWFDBD6EXT5BISHUSKQTM74ZLDZBLYJNTXDEQ, seg_pos={0 0}, upload=1482]: Done uploading piece. err: <nil> Jul 26 23:28:10.285814358 (1h36m26.066747734s) [noise=true , address=45.91.116.6:4973 , piece_id=ES2AH3CA4DB72HPL3GZOOCSL254GK4LX6FS5UMRJKXXU6U3TVVHA, seg_pos={0 0}, upload=1482]: Done uploading piece. err: <nil> Jul 26 23:28:10.287466478 (1h36m26.068399843s) [noise=true , address=45.91.116.12:4227 , piece_id=42RNXBMHTZPW5NDK2SPBEBGTQUNCQIJC33YQJWCEPBBXY5OWJRYQ, seg_pos={0 0}, upload=1482]: Done uploading piece. err: <nil> Jul 26 23:28:10.330310168 (1h36m26.111243549s) [noise=true , address=45.91.116.14:4191 , piece_id=34ZCJ32D6IJFYF2AQ4C4GCVKPZ3U3L5TITHRU57XDYNONEH223IA, seg_pos={0 0}, upload=1482]: Done uploading piece. err: <nil> Jul 26 23:28:10.701325112 (1h36m26.482258478s) [noise=true , address=45.91.116.5:4298 , piece_id=2TR6DDGKF56VSWNV76UG6JMVKJUDJ6U3TMD5YHRWJB3MPLYQ3KEQ, seg_pos={0 0}, upload=1482]: Done uploading piece. err: <nil> Jul 26 23:28:40.101109755 (1h36m55.882043121s) [noise=true , address=104.36.82.8:4971 , piece_id=YQNSSA6XJ7TRSSFSW3NJV6CGNZDOGU2DWSDHKMDQQB7IVKYJUSUA, seg_pos={0 0}, upload=1482]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 159.223.143.39:57846->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:40.101177805 (1h36m55.882111183s) [noise=true , address=104.36.82.9:4007 , piece_id=BUX7CRWB3WSYLBARFJAPYAZG46532J26OCPMMBW6PFCRUY4YI4YA, seg_pos={0 0}, upload=1482]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:40.101240380 (1h36m55.882173758s) [noise=true , address=104.36.82.15:4259 , piece_id=U7VC6KV5S6OMUMWH3WCG6B463576WKW56Q7PC7GMMF3BMPNRO4WA, seg_pos={0 0}, upload=1482]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: rpc: dial tcp 104.36.82.15:4259: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:40.101296355 (1h36m55.882229726s) [noise=true , address=104.36.82.5:4838 , piece_id=6O7SVTD7JYHBFBRZYZNWBUDZ64GXMHLDPLB7VOCKJMBR3T2ZVYHA, seg_pos={0 0}, upload=1482]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:40.101306465 (1h36m55.882239842s) [noise=true , address=104.36.82.16:4295 , piece_id=B26KO53SRNRXP4M4QTOR6M27BTXQH6LBCWVWQDVJFTV24NNIXDAA, seg_pos={0 0}, upload=1482]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:40.101367345 (1h36m55.882300722s) [noise=true , address=104.36.82.6:4874 , piece_id=63GE366FL34LZEGSMM2ZDR2KXNMNQUVTZSSNSAYKY6YG5CI6NCSQ, seg_pos={0 0}, upload=1482]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:40.101382283 (1h36m55.88231565s) [noise=true , address=104.36.82.13:4187 , piece_id=IMYJAMS4O5AAZBPUPW2OXFT642D2ASXKWNIY4FGNDBNE7XH7NYTQ, seg_pos={0 0}, upload=1482]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:40.101406561 (1h36m55.882339931s) [noise=true , address=104.36.82.3:4765 , piece_id=4ZATPHYSOFLNXRDIWYDLDXBJCT2NUWFKAT33NO6ENZM3U2TNG4FQ, seg_pos={0 0}, upload=1482]: Done uploading piece. err: ecclient: upload failed (node:1b1zhuhfXoTexYZUxKUAdCNhWJD8oGC17uXKRykA3gErEEMPWV, address:104.36.82.3:4765): protocol: rpc: dial tcp 104.36.82.3:4765: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:40.101424346 (1h36m55.882357711s) [noise=true , address=104.36.82.7:4912 , piece_id=Y6GI4JJSUWSYJABJXXFUJR6WHHSLJRZ34TV2DQT52XV3TZ3CTW3A, seg_pos={0 0}, upload=1482]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:49246->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:28:40.101473292 (1h36m55.882406674s) [seg_pos={0 0}, upload=1482]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:28:40.131934110 (1h36m55.912867491s) [upload=1482]: Done uploading segments... Jul 26 23:28:40.131987120 (1h36m55.912920496s) [upload=1482]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.033918661s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 29 out of 60 is failed (threshold=3) [.X........................X......XXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.030616695s FAIL (2025-07-26 23:28:37 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.030644453s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.................................XXX.X....XX.X.X.X.......XXX] duration: 30.001469488s FAIL (2025-07-26 23:28:31 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001495926s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..........X.X..........XX...X...X..XX.X...X..X.......X...X..] duration: 48.784177569s FAIL (2025-07-26 23:28:09 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.784222132s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 28 out of 60 is failed (threshold=3) [..X........................X......XXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.031097825s FAIL (2025-07-26 23:27:53 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.031133388s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 25 out of 60 is failed (threshold=3) [...................................XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.031851213s FAIL (2025-07-26 23:27:42 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== aa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327916538 (1h34m47.10884992s) [seg_pos={0 0}, upload=1450]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:26:31.354126428 (1h34m47.135059805s) [upload=1450]: Done uploading segments... Jul 26 23:26:31.354156676 (1h34m47.135090053s) [upload=1450]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Jul 26 23:27:12.406126641 (1h35m28.187060017s) [upload=1467]: Starting upload Jul 26 23:27:12.406195802 (1h35m28.187129168s) [upload=1467]: Done starting upload Jul 26 23:27:12.406422274 (1h35m28.187355641s) [upload=1467]: Uploading segments... Jul 26 23:27:12.406611296 (1h35m28.187544675s) [upload=1467]: Got next segment. Inline: false Jul 26 23:27:12.406742913 (1h35m28.18767629s) [upload=1467]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:27:12.406751844 (1h35m28.187685223s) [upload=1467]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:27:12.435780043 (1h35m28.216713419s) [seg_pos={0 0}, upload=1467]: Begin upload segment... Jul 26 23:27:12.435828004 (1h35m28.216761373s) [seg_pos={0 0}, upload=1467]: Done begin upload segment. Jul 26 23:27:12.435859205 (1h35m28.216792579s) [noise=true , address=77.90.31.16:4734 , piece_id=IWMWDP6F4C6N3XE45DFKCETMEIM27T3WFNDIZ4WJ5WBJPJ2EEBDQ, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.435894044 (1h35m28.216827429s) [noise=true , address=45.91.116.7:4011 , piece_id=Y5AASEQYGOGYPSFVOIK6ZPXGKFDDF2UJ7UZLICXIJRDPV5MMVPQQ, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.436391448 (1h35m28.217324831s) [noise=true , address=45.91.116.10:4083 , piece_id=EFYVD2SLMPSHHQJRGNT3VKMKQQZV6AW4JX7FC6KQISBOUNRILH5Q, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.436558578 (1h35m28.217491964s) [noise=true , address=77.90.31.12:4590 , piece_id=SXRPOVG2CDNAJACM6SOSBNW42TMLXPC5ZLJ3FAEEGG6PMPEQKMXQ, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.436786676 (1h35m28.21772006s) [noise=true , address=45.91.116.5:4298 , piece_id=IGDSSBE6WGYYUYMQQ5U66P5FI3LQ7JANJOWZJR2SMUTYMLIF4M7Q, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.437151469 (1h35m28.218084847s) [noise=true , address=104.36.82.5:4839 , piece_id=RZNCJ7M2X7VJDJWJBRLG4GE24J4OZ2JEVWJ4YMZBCYBQRDKZPZTA, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.437484548 (1h35m28.218417927s) [noise=true , address=45.91.116.13:4153 , piece_id=ANER425M4SOCJUMQEWD742OF4JZJUOLEYP6HHPBR6LFH5MSKXBQA, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.437820365 (1h35m28.218753743s) [noise=true , address=104.36.82.4:4801 , piece_id=CM7DVF77TXSIWLSLQLZ56FUGXKVB2GNO7GLAMYSTD6PPDO6BS2DQ, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.437932378 (1h35m28.218865764s) [noise=true , address=45.91.116.12:4225 , piece_id=FJTWSVHCWCO4CP5NOZI6A7LF5K2ZLBJQDCE2XNUYQXMDZCXG55SA, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.438370932 (1h35m28.219304321s) [noise=true , address=104.36.82.10:4079 , piece_id=Q4PVG5RZNVXHE7AWTNMNJJLYFAPHZCO7IUPIDCPJDXVBKQK4GJCA, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.438793254 (1h35m28.219726639s) [noise=true , address=104.36.82.13:4186 , piece_id=2Q62YQEDO3RD52IP72JQMMR7ZBANB4VOWSHGPI32KCPHDN6VDHCA, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.439124793 (1h35m28.220058176s) [noise=true , address=45.91.116.6:4975 , piece_id=IFV5TOWWFA7TQ3ZAGPKJRJYL44CV3USE5KPTDR2RU22HWNC5EKBQ, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.439515696 (1h35m28.220449074s) [noise=true , address=104.36.82.9:4005 , piece_id=3GYIPBREPTS3H7OVRC4QLG2EKXJFCDGPBLEQYGTDMU4GOVLXARFQ, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.439912524 (1h35m28.220845903s) [noise=true , address=45.91.116.8:4048 , piece_id=PSJ2DXQO25FDASVOV33ZNOUMVZTXINTVUWOCOTRZWPSVW722B2IQ, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.440057808 (1h35m28.220991194s) [noise=true , address=45.91.116.12:4226 , piece_id=WUSEEGGCESNRVCX7QOQNMFKVJTCAO2XNAH22VFMAZOS4BKW6CBJA, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.440587514 (1h35m28.221520964s) [noise=true , address=45.91.116.10:4081 , piece_id=OQNVHOOVR7K2Q5U7SNU3SDJMEJDFINCAF67YOYFJJA6XHTBUK6BQ, seg_pos={0 0}, upload=1467]: Uploading piece... Jul 26 23:27:12.546057090 (1h35m28.32699049s) [upload=1467]: Next returned nil segment Jul 26 23:27:12.546098741 (1h35m28.327032119s) [upload=1467]: Waiting for error group managing segments... Jul 26 23:27:12.548967636 (1h35m28.329901014s) [noise=true , address=45.91.116.10:4081 , piece_id=OQNVHOOVR7K2Q5U7SNU3SDJMEJDFINCAF67YOYFJJA6XHTBUK6BQ, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:12.549616717 (1h35m28.330550095s) [noise=true , address=45.91.116.13:4153 , piece_id=ANER425M4SOCJUMQEWD742OF4JZJUOLEYP6HHPBR6LFH5MSKXBQA, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:12.550171041 (1h35m28.331104411s) [noise=true , address=45.91.116.7:4011 , piece_id=Y5AASEQYGOGYPSFVOIK6ZPXGKFDDF2UJ7UZLICXIJRDPV5MMVPQQ, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:12.550732851 (1h35m28.331666218s) [noise=true , address=45.91.116.12:4225 , piece_id=FJTWSVHCWCO4CP5NOZI6A7LF5K2ZLBJQDCE2XNUYQXMDZCXG55SA, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:12.551402957 (1h35m28.332336323s) [noise=true , address=45.91.116.6:4975 , piece_id=IFV5TOWWFA7TQ3ZAGPKJRJYL44CV3USE5KPTDR2RU22HWNC5EKBQ, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:12.552146469 (1h35m28.333079845s) [noise=true , address=45.91.116.12:4226 , piece_id=WUSEEGGCESNRVCX7QOQNMFKVJTCAO2XNAH22VFMAZOS4BKW6CBJA, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:12.552711637 (1h35m28.333645002s) [noise=true , address=45.91.116.10:4083 , piece_id=EFYVD2SLMPSHHQJRGNT3VKMKQQZV6AW4JX7FC6KQISBOUNRILH5Q, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:12.553390254 (1h35m28.334323618s) [noise=true , address=45.91.116.8:4048 , piece_id=PSJ2DXQO25FDASVOV33ZNOUMVZTXINTVUWOCOTRZWPSVW722B2IQ, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:12.980926756 (1h35m28.761860133s) [noise=true , address=45.91.116.5:4298 , piece_id=IGDSSBE6WGYYUYMQQ5U66P5FI3LQ7JANJOWZJR2SMUTYMLIF4M7Q, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:13.950279768 (1h35m29.731213138s) [noise=true , address=77.90.31.12:4590 , piece_id=SXRPOVG2CDNAJACM6SOSBNW42TMLXPC5ZLJ3FAEEGG6PMPEQKMXQ, seg_pos={0 0}, upload=1467]: Done uploading piece. err: <nil> Jul 26 23:27:42.373490982 (1h35m58.154424358s) [noise=true , address=77.90.31.16:4734 , piece_id=IWMWDP6F4C6N3XE45DFKCETMEIM27T3WFNDIZ4WJ5WBJPJ2EEBDQ, seg_pos={0 0}, upload=1467]: Done uploading piece. err: ecclient: upload failed (node:1srnb2e1ZCtNSpkLsFuMNR6BMdYzH3tqfxZii3qwBnRhbXcee9, address:77.90.31.16:4734): protocol: rpc: dial tcp 77.90.31.16:4734: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:42.373560747 (1h35m58.154494123s) [noise=true , address=104.36.82.13:4186 , piece_id=2Q62YQEDO3RD52IP72JQMMR7ZBANB4VOWSHGPI32KCPHDN6VDHCA, seg_pos={0 0}, upload=1467]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:53166->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:42.373675318 (1h35m58.154608694s) [noise=true , address=104.36.82.10:4079 , piece_id=Q4PVG5RZNVXHE7AWTNMNJJLYFAPHZCO7IUPIDCPJDXVBKQK4GJCA, seg_pos={0 0}, upload=1467]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:27:42.373712909 (1h35m58.154646281s) [noise=true , address=104.36.82.4:4801 , piece_id=CM7DVF77TXSIWLSLQLZ56FUGXKVB2GNO7GLAMYSTD6PPDO6BS2DQ, seg_pos={0 0}, upload=1467]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: write tcp 159.223.143.39:51702->104.36.82.4:4801: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:42.373758162 (1h35m58.154691523s) [noise=true , address=104.36.82.9:4005 , piece_id=3GYIPBREPTS3H7OVRC4QLG2EKXJFCDGPBLEQYGTDMU4GOVLXARFQ, seg_pos={0 0}, upload=1467]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:42.373788016 (1h35m58.154721391s) [noise=true , address=104.36.82.5:4839 , piece_id=RZNCJ7M2X7VJDJWJBRLG4GE24J4OZ2JEVWJ4YMZBCYBQRDKZPZTA, seg_pos={0 0}, upload=1467]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:57326->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:42.373838032 (1h35m58.154771411s) [seg_pos={0 0}, upload=1467]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:27:42.403337269 (1h35m58.184270635s) [upload=1467]: Done uploading segments... Jul 26 23:27:42.403363901 (1h35m58.184297279s) [upload=1467]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.031878197s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 47 out of 60 is failed (threshold=3) [XXXX..XXX.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXX] duration: 1m0.035368223s FAIL (2025-07-26 23:27:39 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== ... Jul 26 23:26:39.659234999 (1h34m55.440168374s) [noise=true , address=104.36.82.55:4202 , piece_id=YPXNX3S5PSYK5DRNZYKWMRIHEAFP7FCLBNDTNAJJ27MYPGIIGRFQ, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.659344193 (1h34m55.440277572s) [noise=true , address=64.49.13.154:4927 , piece_id=6Z3LSXHIX3JXS2LZCQQ3ABUOR55LEQ3OKVYIIDYOOO53KI7GLELQ, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.659481433 (1h34m55.440414807s) [noise=true , address=104.36.82.54:4189 , piece_id=PK7G3SJ7UEUNRXJIEFIKCXUQSDESHJL372Z7LYITW7I42MXUAZKA, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.659732007 (1h34m55.440665385s) [noise=true , address=104.36.82.52:4111 , piece_id=UAUCOWACA6QYBBM7W3TIZFJ3W4SFHCLXWXCWZS5L6TOJBCRPUOZQ, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.659747536 (1h34m55.44068091s) [noise=true , address=77.90.31.15:4712 , piece_id=UFKXB7ETSKH32EXYABNBAINZSPBW5A5MFZYEVFTOG5CE2SF67GDQ, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.659786218 (1h34m55.440719593s) [noise=true , address=104.36.82.47:4527 , piece_id=MK524VHV57PBATOQH7GW4MUOFCCWNUNP2PYH5UBE64XOTZLUZLFA, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.660125196 (1h34m55.441058574s) [noise=true , address=77.90.31.17:4786 , piece_id=YHNOCPEHXANRHEEBOD4KWD7FZ2Z2BYZC2DE5Q5XHZXYA6F43KRSQ, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.660286927 (1h34m55.441220311s) [noise=true , address=79.127.201.215:4346 , piece_id=2XR6CUELXRNQEHTXTII54HES7VPEKIK77WKQRKZQBYBTEHQ7HAKA, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.660472628 (1h34m55.441406007s) [noise=true , address=79.127.219.38:4006 , piece_id=BQTVRTTKNOREACEFSYARXR3SHKBVBE3ICZAOBMAYHOPCBA464EFA, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.660652734 (1h34m55.441586112s) [noise=true , address=79.127.205.246:4280 , piece_id=XGJ5PFEESTFNKRRQRBOF2FBGB3YWZ7CSO7MVOKIBIBSJWYH4ZQYQ, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.660790181 (1h34m55.441723562s) [noise=true , address=156.146.43.225:4448 , piece_id=GV4BCGDA4X6IG47A7POYPRPP3IIGCAXO34DZDJMVJDIVY625Z3FA, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.661118495 (1h34m55.44205187s) [noise=true , address=104.36.82.56:4241 , piece_id=6ZEUOUVT7HE66BO65BMGZR7TASSPIRT6O3PNDNTBDPD33AIMZ6SQ, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.661722162 (1h34m55.44265553s) [noise=true , address=104.36.82.12:4154 , piece_id=JSQ3DG4P4OMLM3WGADASSV2FBULSRBRRQ6D63FOZ723Y2AG2UKBQ, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.664110323 (1h34m55.445043698s) [noise=true , address=77.90.31.19:4844 , piece_id=B7YNOT7HXKFTYYCPTJ5ZAZYEDTFF745V6TMT7GOLAV6IJSRQQBVQ, seg_pos={0 0}, upload=1460]: Uploading piece... Jul 26 23:26:39.697218787 (1h34m55.478152164s) [noise=true , address=77.90.31.13:4634 , piece_id=DUPUJDZPB3Y45BFV3ARYZCGEMID7SI5GE2YABVBQARKD6O2I2YJQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:1WwFU9dJW2LLdTQ64FLEpt4qyQLVuiEb8PmSLrUokav2HGVCw7, address:77.90.31.13:4634): protocol: storage node overloaded, request limit: 2000 Jul 26 23:26:39.736192088 (1h34m55.517125462s) [upload=1460]: Next returned nil segment Jul 26 23:26:39.736209892 (1h34m55.517143259s) [upload=1460]: Waiting for error group managing segments... Jul 26 23:26:39.740390866 (1h34m55.521324233s) [noise=true , address=79.127.219.38:4006 , piece_id=BQTVRTTKNOREACEFSYARXR3SHKBVBE3ICZAOBMAYHOPCBA464EFA, seg_pos={0 0}, upload=1460]: Done uploading piece. err: <nil> Jul 26 23:26:39.777614590 (1h34m55.558547952s) [noise=true , address=77.90.31.17:4786 , piece_id=YHNOCPEHXANRHEEBOD4KWD7FZ2Z2BYZC2DE5Q5XHZXYA6F43KRSQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: <nil> Jul 26 23:26:40.631541532 (1h34m56.412474916s) [noise=true , address=79.127.205.246:4280 , piece_id=XGJ5PFEESTFNKRRQRBOF2FBGB3YWZ7CSO7MVOKIBIBSJWYH4ZQYQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: <nil> Jul 26 23:26:41.210645432 (1h34m56.991578812s) [noise=true , address=156.146.43.225:4448 , piece_id=GV4BCGDA4X6IG47A7POYPRPP3IIGCAXO34DZDJMVJDIVY625Z3FA, seg_pos={0 0}, upload=1460]: Done uploading piece. err: <nil> Jul 26 23:26:47.410738393 (1h35m3.191671763s) [noise=true , address=64.49.13.154:4927 , piece_id=6Z3LSXHIX3JXS2LZCQQ3ABUOR55LEQ3OKVYIIDYOOO53KI7GLELQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: <nil> Jul 26 23:26:50.571174910 (1h35m6.35210828s) [noise=true , address=79.127.201.215:4346 , piece_id=2XR6CUELXRNQEHTXTII54HES7VPEKIK77WKQRKZQBYBTEHQ7HAKA, seg_pos={0 0}, upload=1460]: Done uploading piece. err: <nil> Jul 26 23:27:09.836278592 (1h35m25.61721196s) [noise=true , address=77.90.31.19:4844 , piece_id=B7YNOT7HXKFTYYCPTJ5ZAZYEDTFF745V6TMT7GOLAV6IJSRQQBVQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: <nil> Jul 26 23:27:39.587129956 (1h35m55.368063332s) [noise=true , address=77.90.31.15:4712 , piece_id=UFKXB7ETSKH32EXYABNBAINZSPBW5A5MFZYEVFTOG5CE2SF67GDQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:1bfFfSGARVArNEgchuNzxf7j1QzNbq84fuac8TRfzzsuGGCeT8, address:77.90.31.15:4712): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:27:39.587435246 (1h35m55.368368621s) [noise=true , address=104.36.82.54:4189 , piece_id=PK7G3SJ7UEUNRXJIEFIKCXUQSDESHJL372Z7LYITW7I42MXUAZKA, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:13mB9AoQJubhEJAAVqzd4o4eq8AfNvLkdqHTDAPN3UyXXwE4B7, address:104.36.82.54:4189): protocol: write tcp 159.223.143.39:44002->104.36.82.54:4189: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:39.587539095 (1h35m55.36847247s) [noise=true , address=104.36.82.56:4241 , piece_id=6ZEUOUVT7HE66BO65BMGZR7TASSPIRT6O3PNDNTBDPD33AIMZ6SQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:12tDLUhSXGgBTAJ5n3LWRtP53pJfmkUnDu7CG1kmMXbFpSkHwQ8, address:104.36.82.56:4241): protocol: write tcp 159.223.143.39:60448->104.36.82.56:4241: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:39.587595891 (1h35m55.368529268s) [noise=true , address=104.36.82.35:4605 , piece_id=23EFVOTXIL52TEHUBQHTQPI4TMMVLWETGHGUCKO37MHBCQNZYANA, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:12s1YRbSeZsYjBeTEBGb47tpitCD1hHdnC1Ysr2iUDvm1Wh6aDm, address:104.36.82.35:4605): protocol: write tcp 159.223.143.39:43762->104.36.82.35:4605: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:39.587704546 (1h35m55.368637911s) [noise=true , address=104.36.82.52:4111 , piece_id=UAUCOWACA6QYBBM7W3TIZFJ3W4SFHCLXWXCWZS5L6TOJBCRPUOZQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:12krqZcFoBPfLFbLGQUAjPMzHJ4HKqsKPRg6Xap7nuShaEKBumH, address:104.36.82.52:4111): protocol: write tcp 159.223.143.39:55594->104.36.82.52:4111: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:39.587821522 (1h35m55.36875489s) [noise=true , address=104.36.82.12:4154 , piece_id=JSQ3DG4P4OMLM3WGADASSV2FBULSRBRRQ6D63FOZ723Y2AG2UKBQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: write tcp 159.223.143.39:40554->104.36.82.12:4154: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:39.587897400 (1h35m55.368830765s) [noise=true , address=104.36.82.53:4134 , piece_id=LSDNUBJQ2B6VKKMSRNF77PXKE4PEVUS7GZSJ5KWWP3X3RLE3FISA, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:1q1d7zN2HvDrwEjrJEsGpXxx2aMRTiuj4zwedLAxZGhCzuvNgL, address:104.36.82.53:4134): protocol: write tcp 159.223.143.39:47844->104.36.82.53:4134: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:39.587989813 (1h35m55.368923178s) [noise=true , address=104.36.82.46:4488 , piece_id=ZALKVH3EFKB7ZVKJYKLKBKF444KRAQR4ID3PRMMA4XCS674CCKGA, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:104.36.82.46:4488): protocol: write tcp 159.223.143.39:40048->104.36.82.46:4488: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:39.588144771 (1h35m55.369078149s) [noise=true , address=104.36.82.55:4202 , piece_id=YPXNX3S5PSYK5DRNZYKWMRIHEAFP7FCLBNDTNAJJ27MYPGIIGRFQ, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:121ogdaiVGybXEVsS8LDJuJRYXGxb1LCNnGPuwbde96eSEzkTW4, address:104.36.82.55:4202): protocol: write tcp 159.223.143.39:42452->104.36.82.55:4202: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:39.588218463 (1h35m55.369151839s) [noise=true , address=104.36.82.47:4527 , piece_id=MK524VHV57PBATOQH7GW4MUOFCCWNUNP2PYH5UBE64XOTZLUZLFA, seg_pos={0 0}, upload=1460]: Done uploading piece. err: ecclient: upload failed (node:17xmkbFoj3PTJnAygsjRAcfdd17AfSkPP2xdZp68GGPrvfMuU6, address:104.36.82.47:4527): protocol: write tcp 159.223.143.39:39052->104.36.82.47:4527: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:27:39.588332362 (1h35m55.369265747s) [seg_pos={0 0}, upload=1460]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 23:27:39.620797969 (1h35m55.401731345s) [upload=1460]: Done uploading segments... Jul 26 23:27:39.620828459 (1h35m55.401761838s) [upload=1460]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.035395033s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 11 out of 60 is failed (threshold=3) [..................................XXX.X....XX.X.X.X.......XX] duration: 30.000944629s FAIL (2025-07-26 23:27:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00097002s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 27 out of 60 is failed (threshold=3) [...X........................X......XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.053848936s FAIL (2025-07-26 23:27:23 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 4) Duration: 30.053875618s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 10 out of 60 is failed (threshold=3) [...................................XXX.X....XX.X.X.X.......X] duration: 30.001902058s FAIL (2025-07-26 23:26:42 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001926226s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 26 out of 60 is failed (threshold=3) [....X........................X......XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.031751079s FAIL (2025-07-26 23:26:38 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.031811398s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 24 out of 60 is failed (threshold=3) [....................................XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.029376822s FAIL (2025-07-26 23:26:31 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 43 (1h34m17.151802022s) [upload=1450]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:26:01.420285671 (1h34m17.201219062s) [seg_pos={0 0}, upload=1450]: Begin upload segment... Jul 26 23:26:01.420356758 (1h34m17.201290127s) [seg_pos={0 0}, upload=1450]: Done begin upload segment. Jul 26 23:26:01.420395583 (1h34m17.201328968s) [noise=true , address=45.91.116.5:4297 , piece_id=DGHNUNGANOGH7BQ7NAXOJBJJHWZ4NU3IEILVRT7ZLX6YBBXHYQVA, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.420431859 (1h34m17.201365247s) [noise=true , address=104.36.82.9:4007 , piece_id=JNQ3VJURTSKTJKVEXQD6PS5SFXTA527T6UBMXLLGEX56C6X4KHOA, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.420935417 (1h34m17.201868797s) [noise=true , address=77.90.31.14:4661 , piece_id=ALF43VKH43T3ONH475BERD4ACWJ4V5VBJSMBS2QQMMV7MYFGBYJA, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.421103557 (1h34m17.202036941s) [noise=true , address=45.91.116.14:4191 , piece_id=JWWSEQAZUFPSKP4WJGEHGXZQYDO2IL5IBUISFIFYYYABNXLE6WJQ, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.421339083 (1h34m17.202272472s) [noise=true , address=104.36.82.13:4187 , piece_id=RY367W5IWYZWMG36WHF5VVWT3JIGX6GFUDPNB4WAFTVCDDIXL7WQ, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.421621978 (1h34m17.202555376s) [noise=true , address=104.36.82.5:4840 , piece_id=RGMWONKFDOHRKNMXBTV2FIRTXL2GB4O2KHAEFQIRR3SJBWL4AQFA, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.421777483 (1h34m17.202710865s) [noise=true , address=104.36.82.12:4150 , piece_id=TW2VWDGUHVWD3VXUYDCLCBXLVCYRZDLATZB4Y256VN2IK4Y4ZF3Q, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.422159781 (1h34m17.203093162s) [noise=true , address=104.36.82.5:4837 , piece_id=XT6HGSJGLDOX72ZUILJDHCHPJZV6K3XWO6OLWRDDBCMSY2ABUSBQ, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.422211867 (1h34m17.203145254s) [noise=true , address=104.36.82.16:4295 , piece_id=ZXQL5GKPUDDYBN7DTIB4KKVEQJM4PVIFGXNJD3SWFYQS7J4B4BNA, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.422621888 (1h34m17.203555278s) [noise=true , address=45.91.116.8:4048 , piece_id=Z7ITEZEYW63OAFJJJN3BUCH2LOQW6FOJRBRPBUTCWRMJY3ITDCIA, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.422633751 (1h34m17.203567153s) [noise=true , address=104.36.82.13:4186 , piece_id=5ZWKUPNX3HMM3KKFTL4MQZROL4QLLQY35IU4Y2V2GFPNNJ6DK6HA, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.422706245 (1h34m17.203639632s) [noise=true , address=104.36.82.8:4969 , piece_id=I2W3QHNJ2CZBYTPCRGGLWCYFTI7N3SVK2FMOV5PKHQ3Z7PXRSMBQ, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.423096422 (1h34m17.204029798s) [noise=true , address=45.91.116.10:4082 , piece_id=XFQTL7NY3UNWTPVJTNIIHKEDPQPVNCCK2HHUJGEKXYGO6XCSQICQ, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.423146574 (1h34m17.204079954s) [noise=true , address=104.36.82.7:4912 , piece_id=3BTOM6OJUIV2YA3GXNXW2OJ274LLVTXT7P2DBGWSOZNJJCWK2OMA, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.423478700 (1h34m17.204412083s) [noise=true , address=45.91.116.9:4261 , piece_id=NPKADCMCSRZGKPDXYGMBGVXPJ5GD2FQOGR5FPBLYAUEMXZLNM7RQ, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.436569456 (1h34m17.217502836s) [noise=true , address=104.36.82.3:4767 , piece_id=WGKWPPH7TKDSXYMR3IB4BZPE2ROBWMYTLI6DHFJKMRN7GTB3XBHQ, seg_pos={0 0}, upload=1450]: Uploading piece... Jul 26 23:26:01.517386527 (1h34m17.298319894s) [upload=1450]: Next returned nil segment Jul 26 23:26:01.517403712 (1h34m17.298337079s) [upload=1450]: Waiting for error group managing segments... Jul 26 23:26:01.521040588 (1h34m17.30197397s) [noise=true , address=45.91.116.10:4082 , piece_id=XFQTL7NY3UNWTPVJTNIIHKEDPQPVNCCK2HHUJGEKXYGO6XCSQICQ, seg_pos={0 0}, upload=1450]: Done uploading piece. err: <nil> Jul 26 23:26:01.521679172 (1h34m17.302612535s) [noise=true , address=45.91.116.8:4048 , piece_id=Z7ITEZEYW63OAFJJJN3BUCH2LOQW6FOJRBRPBUTCWRMJY3ITDCIA, seg_pos={0 0}, upload=1450]: Done uploading piece. err: <nil> Jul 26 23:26:01.524571978 (1h34m17.305505364s) [noise=true , address=77.90.31.14:4661 , piece_id=ALF43VKH43T3ONH475BERD4ACWJ4V5VBJSMBS2QQMMV7MYFGBYJA, seg_pos={0 0}, upload=1450]: Done uploading piece. err: <nil> Jul 26 23:26:01.551563819 (1h34m17.332497201s) [noise=true , address=45.91.116.14:4191 , piece_id=JWWSEQAZUFPSKP4WJGEHGXZQYDO2IL5IBUISFIFYYYABNXLE6WJQ, seg_pos={0 0}, upload=1450]: Done uploading piece. err: <nil> Jul 26 23:26:01.644260327 (1h34m17.425193705s) [noise=true , address=45.91.116.5:4297 , piece_id=DGHNUNGANOGH7BQ7NAXOJBJJHWZ4NU3IEILVRT7ZLX6YBBXHYQVA, seg_pos={0 0}, upload=1450]: Done uploading piece. err: <nil> Jul 26 23:26:01.987450406 (1h34m17.768383775s) [noise=true , address=45.91.116.9:4261 , piece_id=NPKADCMCSRZGKPDXYGMBGVXPJ5GD2FQOGR5FPBLYAUEMXZLNM7RQ, seg_pos={0 0}, upload=1450]: Done uploading piece. err: <nil> Jul 26 23:26:31.327252944 (1h34m47.10818631s) [noise=true , address=104.36.82.5:4837 , piece_id=XT6HGSJGLDOX72ZUILJDHCHPJZV6K3XWO6OLWRDDBCMSY2ABUSBQ, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: write tcp 159.223.143.39:45308->104.36.82.5:4837: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327344545 (1h34m47.108277935s) [noise=true , address=104.36.82.5:4840 , piece_id=RGMWONKFDOHRKNMXBTV2FIRTXL2GB4O2KHAEFQIRR3SJBWL4AQFA, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:34572->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327415428 (1h34m47.108348803s) [noise=true , address=104.36.82.12:4150 , piece_id=TW2VWDGUHVWD3VXUYDCLCBXLVCYRZDLATZB4Y256VN2IK4Y4ZF3Q, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327467550 (1h34m47.108400926s) [noise=true , address=104.36.82.16:4295 , piece_id=ZXQL5GKPUDDYBN7DTIB4KKVEQJM4PVIFGXNJD3SWFYQS7J4B4BNA, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327566254 (1h34m47.108499629s) [noise=true , address=104.36.82.9:4007 , piece_id=JNQ3VJURTSKTJKVEXQD6PS5SFXTA527T6UBMXLLGEX56C6X4KHOA, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:52264->104.36.82.9:4007: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327659406 (1h34m47.108592781s) [noise=true , address=104.36.82.13:4186 , piece_id=5ZWKUPNX3HMM3KKFTL4MQZROL4QLLQY35IU4Y2V2GFPNNJ6DK6HA, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:44064->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327720925 (1h34m47.108654294s) [noise=true , address=104.36.82.7:4912 , piece_id=3BTOM6OJUIV2YA3GXNXW2OJ274LLVTXT7P2DBGWSOZNJJCWK2OMA, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327799308 (1h34m47.108732672s) [noise=true , address=104.36.82.3:4767 , piece_id=WGKWPPH7TKDSXYMR3IB4BZPE2ROBWMYTLI6DHFJKMRN7GTB3XBHQ, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:57206->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327852056 (1h34m47.108785431s) [noise=true , address=104.36.82.13:4187 , piece_id=RY367W5IWYZWMG36WHF5VVWT3JIGX6GFUDPNB4WAFTVCDDIXL7WQ, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327888899 (1h34m47.108822275s) [noise=true , address=104.36.82.8:4969 , piece_id=I2W3QHNJ2CZBYTPCRGGLWCYFTI7N3SVK2FMOV5PKHQ3Z7PXRSMBQ, seg_pos={0 0}, upload=1450]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:26:31.327916538 (1h34m47.10884992s) [seg_pos={0 0}, upload=1450]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:26:31.354126428 (1h34m47.135059805s) [upload=1450]: Done uploading segments... Jul 26 23:26:31.354156676 (1h34m47.135090053s) [upload=1450]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration:* 30.029407026s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)