Orbiter History

checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [X......X.XX....X...X.................XX....X......XX.XXXX...] duration: 1.941315999s FAIL (2025-07-27 00:37:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.941345054s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029120071s FAIL (2025-07-27 00:37:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== ing piece... Jul 27 00:36:43.767932742 (2h44m59.548866121s) [noise=true , address=104.36.82.42:4821 , piece_id=UYVX2TUJ7WV5IYW2D357U7ZRQZTU5ZXRMWFOQNBSCHZHFWJHJWPA, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.768026314 (2h44m59.548959693s) [noise=true , address=79.127.201.214:4321 , piece_id=ITWO3CXXN5NDC7PG6XEYINFLRQ6SGQDS5MM5MLCG35CHS6XOWT5Q, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.768368082 (2h44m59.549301459s) [noise=true , address=104.36.82.47:4522 , piece_id=TYMDJXNZRAFTXNRNNBVA2KJ65ZZS5RGBAC7BDPXM2HSJQJ364T3A, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.768435991 (2h44m59.54936937s) [noise=true , address=104.36.82.4:4826 , piece_id=GOLCJ7JSJ6OCQUKWPM6F5EV7PGY2HNBSP2DLBXGITRUXYPW6KZNQ, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.768699993 (2h44m59.549633374s) [noise=true , address=45.91.116.11:4130 , piece_id=GENZ6MSX2L4VM5LXHE27PEKIB76GIEQK2UYHKKB37QBAW42PHUEA, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.768842307 (2h44m59.549775676s) [noise=true , address=79.127.205.235:4687 , piece_id=WJX7AZPSMRF7WUM3LCEDKQJUC6L5TSCSHYAIG75RFRDDLFTQO4YQ, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.768995272 (2h44m59.549928654s) [noise=true , address=77.90.31.15:4720 , piece_id=BJQZAPN2VQGI4UZBODREWLRZB57ZCK37DZYROY6GLNDCBAQEUUZQ, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.769273110 (2h44m59.550206485s) [noise=true , address=104.36.82.46:4515 , piece_id=R7ZZDKNGNMQ2BZWDJUUNXUFYXGTYMR427DZ7ABW6SVKBPDJDJXSQ, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.769555178 (2h44m59.550488553s) [noise=true , address=79.127.220.101:4041 , piece_id=USHOT7M2CM7GLICKIQBKB7R2VFQQ7OK5PHG2FS575EL2HHKRRQ3Q, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.769851605 (2h44m59.550784991s) [noise=true , address=64.49.13.137:4502 , piece_id=CDW7WFPA6NBYILJYFUVG3X5SCHRX364RAF7KIRRA62WBONXCXUJQ, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.770183087 (2h44m59.551116465s) [noise=true , address=45.91.116.10:4093 , piece_id=6HKNOJ3ZJL3YONQYAGPPTDZ64L7YO5G6FBWGEFAORNWG5NHXGYVA, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.770520905 (2h44m59.551454283s) [noise=true , address=45.91.116.9:4277 , piece_id=S36THBV2H7SULXMWK6TDJJTLV2K2I2GJGRR7KBVSFQJCKHTRDJ3Q, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.770886534 (2h44m59.551819914s) [noise=true , address=77.90.31.10:4523 , piece_id=WO5HKLFIJ6JDWMGXDSYEGRHP45KEHOOW5ZEMCI66PUTLAIWOOJAA, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.771105024 (2h44m59.552038406s) [noise=true , address=104.36.82.56:4231 , piece_id=XGDAWE76KCVZ5FML6ASX4SVQCE5FM2UNARDPPEO3RDLWS7CJBGLA, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.771498289 (2h44m59.552431791s) [noise=true , address=104.36.82.7:4926 , piece_id=OKARRYY5ZUAZFQ7QJLFYJTNMQA7VAIXMXZADI6ZMZIIWCZHNJQWA, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.771990555 (2h44m59.552923936s) [noise=true , address=79.127.205.234:4612 , piece_id=KFJ7XADVEY64W2M74O2FSHS6HV7GBRLUQENA6MIJXLNS5OUTP6DQ, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.772401408 (2h44m59.553334794s) [noise=true , address=79.127.201.215:4349 , piece_id=RYGHCEURYGLJKWWNSRGIGZSRNBTE55NP335CGMPLLETIKCMBCCMQ, seg_pos={0 0}, upload=2535]: Uploading piece... Jul 27 00:36:43.871311119 (2h44m59.652244485s) [upload=2535]: Next returned nil segment Jul 27 00:36:43.871346890 (2h44m59.652280273s) [upload=2535]: Waiting for error group managing segments... Jul 27 00:36:44.351148542 (2h45m0.132081922s) [noise=true , address=79.127.205.234:4612 , piece_id=KFJ7XADVEY64W2M74O2FSHS6HV7GBRLUQENA6MIJXLNS5OUTP6DQ, seg_pos={0 0}, upload=2535]: Done uploading piece. err: <nil> Jul 27 00:36:44.386342709 (2h45m0.167276094s) [noise=true , address=79.127.205.235:4687 , piece_id=WJX7AZPSMRF7WUM3LCEDKQJUC6L5TSCSHYAIG75RFRDDLFTQO4YQ, seg_pos={0 0}, upload=2535]: Done uploading piece. err: <nil> Jul 27 00:36:45.046538362 (2h45m0.827471727s) [noise=true , address=64.49.13.137:4502 , piece_id=CDW7WFPA6NBYILJYFUVG3X5SCHRX364RAF7KIRRA62WBONXCXUJQ, seg_pos={0 0}, upload=2535]: Done uploading piece. err: <nil> Jul 27 00:36:47.724316954 (2h45m3.505250323s) [noise=true , address=79.127.201.214:4321 , piece_id=ITWO3CXXN5NDC7PG6XEYINFLRQ6SGQDS5MM5MLCG35CHS6XOWT5Q, seg_pos={0 0}, upload=2535]: Done uploading piece. err: <nil> Jul 27 00:36:49.376750178 (2h45m5.157683547s) [noise=true , address=45.91.116.11:4130 , piece_id=GENZ6MSX2L4VM5LXHE27PEKIB76GIEQK2UYHKKB37QBAW42PHUEA, seg_pos={0 0}, upload=2535]: Done uploading piece. err: ecclient: upload failed (node:126ThYkfUAYf2JR8f5rkwyumcvmRArxKbpLUjhJukoc3FADHj2T, address:45.91.116.11:4130): protocol: storage node overloaded, request limit: 2000 Jul 27 00:36:50.355177364 (2h45m6.136110743s) [noise=true , address=79.127.220.101:4041 , piece_id=USHOT7M2CM7GLICKIQBKB7R2VFQQ7OK5PHG2FS575EL2HHKRRQ3Q, seg_pos={0 0}, upload=2535]: Done uploading piece. err: <nil> Jul 27 00:36:51.429978417 (2h45m7.210911787s) [noise=true , address=45.91.116.10:4093 , piece_id=6HKNOJ3ZJL3YONQYAGPPTDZ64L7YO5G6FBWGEFAORNWG5NHXGYVA, seg_pos={0 0}, upload=2535]: Done uploading piece. err: <nil> Jul 27 00:36:57.236828201 (2h45m13.017761569s) [noise=true , address=45.91.116.9:4277 , piece_id=S36THBV2H7SULXMWK6TDJJTLV2K2I2GJGRR7KBVSFQJCKHTRDJ3Q, seg_pos={0 0}, upload=2535]: Done uploading piece. err: <nil> Jul 27 00:37:01.658388202 (2h45m17.439321582s) [noise=true , address=79.127.201.215:4349 , piece_id=RYGHCEURYGLJKWWNSRGIGZSRNBTE55NP335CGMPLLETIKCMBCCMQ, seg_pos={0 0}, upload=2535]: Done uploading piece. err: <nil> Jul 27 00:37:23.872581318 (2h45m39.653514688s) [noise=true , address=77.90.31.10:4523 , piece_id=WO5HKLFIJ6JDWMGXDSYEGRHP45KEHOOW5ZEMCI66PUTLAIWOOJAA, seg_pos={0 0}, upload=2535]: Done uploading piece. err: <nil> Jul 27 00:37:43.695477686 (2h45m59.476411062s) [noise=true , address=104.36.82.7:4926 , piece_id=OKARRYY5ZUAZFQ7QJLFYJTNMQA7VAIXMXZADI6ZMZIIWCZHNJQWA, seg_pos={0 0}, upload=2535]: Done uploading piece. err: ecclient: upload failed (node:12EXjvaPQL9GhezbMgqhQBZvEKk9UTEMp8n8G2Pet5F3r4XLpDr, address:104.36.82.7:4926): protocol: rpc: dial tcp 104.36.82.7:4926: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:43.695508733 (2h45m59.476442109s) [noise=true , address=104.36.82.47:4522 , piece_id=TYMDJXNZRAFTXNRNNBVA2KJ65ZZS5RGBAC7BDPXM2HSJQJ364T3A, seg_pos={0 0}, upload=2535]: Done uploading piece. err: ecclient: upload failed (node:1zGZe6e37uop4y622fU1GbspMPVEZURS3FbMrf7je6VT9TYGFr, address:104.36.82.47:4522): protocol: write tcp 159.223.143.39:54132->104.36.82.47:4522: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:43.695546438 (2h45m59.476479815s) [noise=true , address=104.36.82.15:4277 , piece_id=QT3D4NZQANATTR6RBNY57NDVAKZ257LY7PFNFKGUWT6NLSPUPUFA, seg_pos={0 0}, upload=2535]: Done uploading piece. err: ecclient: upload failed (node:1i493KQZuQYQwXzct9HamWDCq92CSGvLPHjZ6fxxVVZX1f7ofY, address:104.36.82.15:4277): protocol: rpc: dial tcp 104.36.82.15:4277: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:43.695603803 (2h45m59.476537169s) [noise=true , address=104.36.82.46:4515 , piece_id=R7ZZDKNGNMQ2BZWDJUUNXUFYXGTYMR427DZ7ABW6SVKBPDJDJXSQ, seg_pos={0 0}, upload=2535]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: write tcp 159.223.143.39:43322->104.36.82.46:4515: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:43.695614733 (2h45m59.476548113s) [noise=true , address=104.36.82.56:4231 , piece_id=XGDAWE76KCVZ5FML6ASX4SVQCE5FM2UNARDPPEO3RDLWS7CJBGLA, seg_pos={0 0}, upload=2535]: Done uploading piece. err: ecclient: upload failed (node:129WWyyn7SrJFbFJ2kjPVhwDNRoYM7Kj449TZiZLzUbqpwK7qJ4, address:104.36.82.56:4231): protocol: rpc: dial tcp 104.36.82.56:4231: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:43.695669453 (2h45m59.476602838s) [noise=true , address=104.36.82.4:4826 , piece_id=GOLCJ7JSJ6OCQUKWPM6F5EV7PGY2HNBSP2DLBXGITRUXYPW6KZNQ, seg_pos={0 0}, upload=2535]: Done uploading piece. err: ecclient: upload failed (node:14hpQXA7AA3CQQ4z8Y81ZTxjY9Bhd1eNCjqEKyandGUHoyi548, address:104.36.82.4:4826): protocol: rpc: dial tcp 104.36.82.4:4826: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:43.695807449 (2h45m59.476740815s) [noise=true , address=77.90.31.15:4720 , piece_id=BJQZAPN2VQGI4UZBODREWLRZB57ZCK37DZYROY6GLNDCBAQEUUZQ, seg_pos={0 0}, upload=2535]: Done uploading piece. err: ecclient: upload failed (node:12VbsxFqMB3vVphSAJqYpbBzua7Goq45E6ovJSaExkpLtfkcc74, address:77.90.31.15:4720): protocol: write tcp 159.223.143.39:52870->77.90.31.15:4720: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:43.695902245 (2h45m59.476835626s) [noise=true , address=104.36.82.42:4821 , piece_id=UYVX2TUJ7WV5IYW2D357U7ZRQZTU5ZXRMWFOQNBSCHZHFWJHJWPA, seg_pos={0 0}, upload=2535]: Done uploading piece. err: ecclient: upload failed (node:1Q2vaBr3JNY3jv8jVUTg9uzfWUtNjrd5c5Z9Yfi54D3JMRM1af, address:104.36.82.42:4821): protocol: rpc: dial tcp 104.36.82.42:4821: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 27 00:37:43.695961267 (2h45m59.47689465s) [seg_pos={0 0}, upload=2535]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:37:43.721488514 (2h45m59.502421881s) [upload=2535]: Done uploading segments... Jul 27 00:37:43.721531714 (2h45m59.502465083s) [upload=2535]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.029140824s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.............................................X..............] duration: 248.958156ms FAIL (2025-07-27 00:37:39 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 248.984515ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.....X...XX...............................X...............X.] duration: 51.197774262s FAIL (2025-07-27 00:37:39 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 51.197786566s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXXX] duration: 30.046126676s FAIL (2025-07-27 00:37:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.046172758s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X..X] duration: 30.026916589s FAIL (2025-07-27 00:37:19 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== conn closed: op Jul 27 00:35:58.912470783 (2h44m14.693404168s) [seg_pos={0 0}, upload=2525]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:35:58.912556900 (2h44m14.693490281s) [upload=2525]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:35:58.912562580 (2h44m14.693495961s) [upload=2525]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:35:58.959169793 (2h44m14.740103173s) [upload=2525]: Done uploading segments... Jul 27 00:35:58.959189338 (2h44m14.740122714s) [upload=2525]: Upload finished. err: <nil> Jul 27 00:36:49.246831627 (2h45m5.027765004s) [upload=2538]: Starting upload Jul 27 00:36:49.246924215 (2h45m5.027857599s) [upload=2538]: Done starting upload Jul 27 00:36:49.247146644 (2h45m5.028080026s) [upload=2538]: Uploading segments... Jul 27 00:36:49.247418538 (2h45m5.028351922s) [upload=2538]: Got next segment. Inline: false Jul 27 00:36:49.247493810 (2h45m5.028427181s) [upload=2538]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:36:49.247500601 (2h45m5.028433979s) [upload=2538]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:36:49.361431565 (2h45m5.142364941s) [seg_pos={0 0}, upload=2538]: Begin upload segment... Jul 27 00:36:49.361486677 (2h45m5.142420053s) [seg_pos={0 0}, upload=2538]: Done begin upload segment. Jul 27 00:36:49.361510789 (2h45m5.142444172s) [noise=true , address=45.91.116.6:4974 , piece_id=XONYL2PYKZUMEKZE4W7WRTV6IVUYFPGBSPSBAFMTI6UH2WTRM2VA, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.361525521 (2h45m5.142458909s) [noise=true , address=104.36.82.8:4972 , piece_id=UWC46P6MZRKO336XASYCYE4DNL2EL2GOPTVVFRQHYQHPWUKUHTGA, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.361990020 (2h45m5.142923422s) [noise=true , address=77.90.31.11:4554 , piece_id=4TDYCFCSOHMP7VXXX7PXR2PI5CBXIKT5TYRNKLXZ4FHJ7CX7YA3A, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.362114146 (2h45m5.143047535s) [noise=true , address=104.36.82.9:4007 , piece_id=LZBMPOBSL44MAUVUNGRQ5GIBCL6TXMDRHPY73MCNLITKGECE52UA, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.362522374 (2h45m5.143455752s) [noise=true , address=77.90.31.10:4517 , piece_id=W36AMIHXFO33E2D5QNFJVTCQXPJNUX5GALTSLYLDMHW3D574UHAQ, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.362623719 (2h45m5.143557098s) [noise=true , address=45.91.116.5:4297 , piece_id=ZCLLHQOPS3QSUPXBUW5HM2XHCMXRIYADMVFKNB6Q35X5V64R3GIQ, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.363044271 (2h45m5.143977648s) [noise=true , address=77.90.31.15:4698 , piece_id=R7BVEFUKRNJARZVBW3WGQVE25STNEFPWUXGINXDEIVWTHLNTJX5Q, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.363105102 (2h45m5.144038501s) [noise=true , address=45.91.116.4:4937 , piece_id=B2HCWW4ICKF72LKQ62NUTBUPG4CN6IGTOUWNHA34FCQYUP5O2KTQ, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.363557375 (2h45m5.144490758s) [noise=true , address=45.91.116.9:4261 , piece_id=GGERWB3IFPNWJ7KME3OF7GKKBJPEFCB62XXV2OOAP4FZW6TYWK6Q, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.364143926 (2h45m5.145077308s) [noise=true , address=77.90.31.14:4661 , piece_id=4UUM63CXD6GGHH2PU3EXEY3576PPE6B7YNA4G6HRNLDD3R7DMSXA, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.364549548 (2h45m5.145482924s) [noise=true , address=104.36.82.5:4840 , piece_id=EWHHPZZG7NHPRMEBQOQCDC3A46HX4ZHWR5HO4J4GM2KSRUXAAOFA, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.364991231 (2h45m5.145924614s) [noise=true , address=104.36.82.8:4971 , piece_id=IBJI6TBLB5O5C3NSAFCFOJHJN25JHVT47QRLDFNKHSUZSBQXJUSA, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.365424115 (2h45m5.146357502s) [noise=true , address=77.90.31.17:4771 , piece_id=3BV5T2W777IZB7PPU5Z737MRKKX2XMSAVAC4HJPXU5MPU2B72RRQ, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.365821330 (2h45m5.146754714s) [noise=true , address=45.91.116.10:4081 , piece_id=GEGWUVKULKCNRRZW4OYWG42ST3HHVKGZNFJLIDFRTSC6R3RF3JTQ, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.366327018 (2h45m5.147260405s) [noise=true , address=45.91.116.7:4011 , piece_id=MGHVYRPMHVELCN7MCVS5YGJTB46EE56GZTNJBB5UJ4ANIRSI3FUQ, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.366885702 (2h45m5.147819094s) [noise=true , address=104.36.82.5:4839 , piece_id=EZIZKT3FTJW6YX4OKQE5JUOUPO6NFN3INKN7UZYZ4RDFJJF376YQ, seg_pos={0 0}, upload=2538]: Uploading piece... Jul 27 00:36:49.486871844 (2h45m5.267805231s) [upload=2538]: Next returned nil segment Jul 27 00:36:49.486894239 (2h45m5.267827627s) [upload=2538]: Waiting for error group managing segments... Jul 27 00:36:49.491705522 (2h45m5.272638913s) [noise=true , address=77.90.31.17:4771 , piece_id=3BV5T2W777IZB7PPU5Z737MRKKX2XMSAVAC4HJPXU5MPU2B72RRQ, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:36:49.492059404 (2h45m5.272992785s) [noise=true , address=45.91.116.7:4011 , piece_id=MGHVYRPMHVELCN7MCVS5YGJTB46EE56GZTNJBB5UJ4ANIRSI3FUQ, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:36:49.492530690 (2h45m5.273464059s) [noise=true , address=45.91.116.4:4937 , piece_id=B2HCWW4ICKF72LKQ62NUTBUPG4CN6IGTOUWNHA34FCQYUP5O2KTQ, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:36:49.492789433 (2h45m5.273722812s) [noise=true , address=45.91.116.9:4261 , piece_id=GGERWB3IFPNWJ7KME3OF7GKKBJPEFCB62XXV2OOAP4FZW6TYWK6Q, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:36:49.492936408 (2h45m5.273869793s) [noise=true , address=45.91.116.6:4974 , piece_id=XONYL2PYKZUMEKZE4W7WRTV6IVUYFPGBSPSBAFMTI6UH2WTRM2VA, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:36:49.493417226 (2h45m5.274350598s) [noise=true , address=45.91.116.10:4081 , piece_id=GEGWUVKULKCNRRZW4OYWG42ST3HHVKGZNFJLIDFRTSC6R3RF3JTQ, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:36:49.493578680 (2h45m5.27451206s) [noise=true , address=77.90.31.14:4661 , piece_id=4UUM63CXD6GGHH2PU3EXEY3576PPE6B7YNA4G6HRNLDD3R7DMSXA, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:36:49.493775828 (2h45m5.274709194s) [noise=true , address=45.91.116.5:4297 , piece_id=ZCLLHQOPS3QSUPXBUW5HM2XHCMXRIYADMVFKNB6Q35X5V64R3GIQ, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:36:49.496148643 (2h45m5.277082022s) [noise=true , address=77.90.31.11:4554 , piece_id=4TDYCFCSOHMP7VXXX7PXR2PI5CBXIKT5TYRNKLXZ4FHJ7CX7YA3A, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:36:51.476997991 (2h45m7.25793136s) [noise=true , address=77.90.31.10:4517 , piece_id=W36AMIHXFO33E2D5QNFJVTCQXPJNUX5GALTSLYLDMHW3D574UHAQ, seg_pos={0 0}, upload=2538]: Done uploading piece. err: <nil> Jul 27 00:37:19.217501107 (2h45m34.998434494s) [noise=true , address=104.36.82.5:4839 , piece_id=EZIZKT3FTJW6YX4OKQE5JUOUPO6NFN3INKN7UZYZ4RDFJJF376YQ, seg_pos={0 0}, upload=2538]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:50572->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:19.217623677 (2h45m34.998557053s) [noise=true , address=104.36.82.8:4972 , piece_id=UWC46P6MZRKO336XASYCYE4DNL2EL2GOPTVVFRQHYQHPWUKUHTGA, seg_pos={0 0}, upload=2538]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: rpc: dial tcp 104.36.82.8:4972: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:19.217699165 (2h45m34.998632552s) [noise=true , address=104.36.82.9:4007 , piece_id=LZBMPOBSL44MAUVUNGRQ5GIBCL6TXMDRHPY73MCNLITKGECE52UA, seg_pos={0 0}, upload=2538]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:19.217742574 (2h45m34.998675953s) [noise=true , address=104.36.82.5:4840 , piece_id=EWHHPZZG7NHPRMEBQOQCDC3A46HX4ZHWR5HO4J4GM2KSRUXAAOFA, seg_pos={0 0}, upload=2538]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: rpc: dial tcp 104.36.82.5:4840: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:19.217791564 (2h45m34.998724951s) [noise=true , address=104.36.82.8:4971 , piece_id=IBJI6TBLB5O5C3NSAFCFOJHJN25JHVT47QRLDFNKHSUZSBQXJUSA, seg_pos={0 0}, upload=2538]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:19.217838532 (2h45m34.998771913s) [noise=true , address=77.90.31.15:4698 , piece_id=R7BVEFUKRNJARZVBW3WGQVE25STNEFPWUXGINXDEIVWTHLNTJX5Q, seg_pos={0 0}, upload=2538]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 159.223.143.39:49862->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:37:19.217865608 (2h45m34.998798994s) [seg_pos={0 0}, upload=2538]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:37:19.241912345 (2h45m35.022845714s) [upload=2538]: Done uploading segments... Jul 27 00:37:19.241938238 (2h45m35.022871613s) [upload=2538]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.026943602s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X......X.XX....X...X.................XX....X......XX.XXXX..] duration: 16.371995472s FAIL (2025-07-27 00:36:55 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 16.372025045s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXXX] duration: 30.04092315s FAIL (2025-07-27 00:36:52 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.040957494s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [......X...XX...............................X...............X] duration: 1m0.001046404s FAIL (2025-07-27 00:36:48 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001065017s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.030475186s FAIL (2025-07-27 00:36:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 633s) [noise=true , address=104.36.82.54:4168 , piece_id=GHOKLEOAY3CXMFBWG4ZPJWWO7KHYHMXSDJ3HLB5D5VHMBVOOLI2Q, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.769593367 (2h43m59.550526753s) [noise=true , address=79.127.219.38:4006 , piece_id=JRIFMNH7U5U5HRT2OLGGIPRNYL6UNV2XEPJE7PF3YOIJMTWIZ6DA, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.770020956 (2h43m59.550954342s) [noise=true , address=79.127.205.246:4277 , piece_id=2K5F6JEVR4Z3PQZAKHXKESAES7HPTBJVPMDURJ7OKKVHQKTQY4KQ, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.770106421 (2h43m59.551039806s) [noise=true , address=77.90.31.17:4801 , piece_id=B4QBFVW6N6SZXZL4R3FPI3KT6HESRE6XUHILSGL6RHAIRBQEVHQQ, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.770435664 (2h43m59.55136905s) [noise=true , address=79.127.205.245:4243 , piece_id=NCKWUZ53ESF5VBR2YUKCNKYXIMRKWMNBFG5UUYOIENTP6IP2IM7Q, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.770525511 (2h43m59.551458898s) [noise=true , address=104.36.82.15:4277 , piece_id=7NW6ZCVVR7ZR3PQ65CBIZENJJM3HRL3GMHGQOHL3UNYRXQP37BDA, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.770823156 (2h43m59.551756543s) [noise=true , address=104.36.82.56:4252 , piece_id=R43YDGDQEUTT4PMDBBA6LHDXUZR4YCBKR43OCD3JEPQI65LQZTZQ, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.770941550 (2h43m59.551874934s) [noise=true , address=104.36.82.44:4886 , piece_id=2AS3FEBOZ2EGVBG3EACVXFKOVE4MDY2CQI4HLPHAQWVMH35DHUPQ, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.771276510 (2h43m59.552209895s) [noise=true , address=104.36.82.46:4502 , piece_id=Q647ENMCYF7F4JWE4YCRY3VGLXOEQ5ZWQJ5CH6EDEDM2KZ6DLD6Q, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.771415114 (2h43m59.552348499s) [noise=true , address=79.127.205.234:4607 , piece_id=AAUUX32NMVEYZMCE4BJ2SBY46WDQQFEYXP62NDVTHTVTYO4LELEA, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.771699733 (2h43m59.552633121s) [noise=true , address=79.127.205.235:4686 , piece_id=ZDUFNQ2MP6JQUW2NUX76TZWSEK62PV5UY5CHY4DWW5YWMYGMAMEA, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.772079512 (2h43m59.553012897s) [noise=true , address=104.36.82.53:4151 , piece_id=OBTWN6PCKL4RUGRM4OBCLDLAAULA2RPDG73I2DHMLKQCXOSY3GCQ, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.772271192 (2h43m59.553204578s) [noise=true , address=79.127.201.215:4355 , piece_id=UOAWKOZ4BIMVDSI4PRGSRIUKFUMXDJQO5BLUVERCM7V2BWHLXGVA, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.772534884 (2h43m59.553468269s) [noise=true , address=79.127.205.233:4653 , piece_id=IKCXDR4LAO6HKKBYI77S4VVNPYDIKMGPZGEAQJ3CLKR23LCC7ZHA, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.772649281 (2h43m59.553582663s) [noise=true , address=77.90.31.11:4569 , piece_id=WCKYUPNVWSAWPVQI5P4W5CGR3UDCUZ2ITCOR7KWLL2IQVTSRFMMA, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.773121589 (2h43m59.554054974s) [noise=true , address=104.36.82.47:4527 , piece_id=K2QVZZMUS4YDQV3XVG7CDESSCDPRIDLDVQOOV33UGVE5JMHE2V6A, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.774109035 (2h43m59.555042417s) [noise=true , address=156.146.43.225:4461 , piece_id=DZF66PYI7MHHJ4WGTOU6GTTFNLSZN7X2HNRZPSIW7PIKNWG6FIPQ, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.774657740 (2h43m59.555591119s) [noise=true , address=45.91.116.14:4206 , piece_id=3MTQ66Z5BO6E6COJRC6W3S5FXWWQHFTCPA6HUSDZ6LFZOFMQGTYA, seg_pos={0 0}, upload=2521]: Uploading piece... Jul 27 00:35:43.868100219 (2h43m59.649033588s) [upload=2521]: Next returned nil segment Jul 27 00:35:43.868116959 (2h43m59.649050323s) [upload=2521]: Waiting for error group managing segments... Jul 27 00:35:43.872695418 (2h43m59.653628785s) [noise=true , address=79.127.219.38:4006 , piece_id=JRIFMNH7U5U5HRT2OLGGIPRNYL6UNV2XEPJE7PF3YOIJMTWIZ6DA, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:35:43.906564137 (2h43m59.687497521s) [noise=true , address=77.90.31.11:4569 , piece_id=WCKYUPNVWSAWPVQI5P4W5CGR3UDCUZ2ITCOR7KWLL2IQVTSRFMMA, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:35:43.964769076 (2h43m59.745702445s) [noise=true , address=77.90.31.17:4801 , piece_id=B4QBFVW6N6SZXZL4R3FPI3KT6HESRE6XUHILSGL6RHAIRBQEVHQQ, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:35:44.069539990 (2h43m59.850473356s) [noise=true , address=156.146.43.225:4461 , piece_id=DZF66PYI7MHHJ4WGTOU6GTTFNLSZN7X2HNRZPSIW7PIKNWG6FIPQ, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:35:44.353630136 (2h44m0.134563501s) [noise=true , address=79.127.205.234:4607 , piece_id=AAUUX32NMVEYZMCE4BJ2SBY46WDQQFEYXP62NDVTHTVTYO4LELEA, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:35:44.393257826 (2h44m0.174191208s) [noise=true , address=79.127.205.235:4686 , piece_id=ZDUFNQ2MP6JQUW2NUX76TZWSEK62PV5UY5CHY4DWW5YWMYGMAMEA, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:35:44.447459336 (2h44m0.228392705s) [noise=true , address=79.127.205.246:4277 , piece_id=2K5F6JEVR4Z3PQZAKHXKESAES7HPTBJVPMDURJ7OKKVHQKTQY4KQ, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:35:44.466280924 (2h44m0.247214293s) [noise=true , address=79.127.205.245:4243 , piece_id=NCKWUZ53ESF5VBR2YUKCNKYXIMRKWMNBFG5UUYOIENTP6IP2IM7Q, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:35:44.601309780 (2h44m0.382243149s) [noise=true , address=79.127.205.233:4653 , piece_id=IKCXDR4LAO6HKKBYI77S4VVNPYDIKMGPZGEAQJ3CLKR23LCC7ZHA, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:35:46.698944176 (2h44m2.479877562s) [noise=true , address=45.91.116.14:4206 , piece_id=3MTQ66Z5BO6E6COJRC6W3S5FXWWQHFTCPA6HUSDZ6LFZOFMQGTYA, seg_pos={0 0}, upload=2521]: Done uploading piece. err: ecclient: upload failed (node:12d12pAMGhxjtLBE1tp9yEE7NMdfh8kcvsA8abhTnM8nbBB5LEK, address:45.91.116.14:4206): protocol: storage node overloaded, request limit: 2000 Jul 27 00:36:02.107058706 (2h44m17.887992087s) [noise=true , address=79.127.201.215:4355 , piece_id=UOAWKOZ4BIMVDSI4PRGSRIUKFUMXDJQO5BLUVERCM7V2BWHLXGVA, seg_pos={0 0}, upload=2521]: Done uploading piece. err: <nil> Jul 27 00:36:43.608681031 (2h44m59.389614416s) [noise=true , address=104.36.82.47:4527 , piece_id=K2QVZZMUS4YDQV3XVG7CDESSCDPRIDLDVQOOV33UGVE5JMHE2V6A, seg_pos={0 0}, upload=2521]: Done uploading piece. err: ecclient: upload failed (node:17xmkbFoj3PTJnAygsjRAcfdd17AfSkPP2xdZp68GGPrvfMuU6, address:104.36.82.47:4527): protocol: rpc: dial tcp 104.36.82.47:4527: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:43.608766466 (2h44m59.389699847s) [noise=true , address=104.36.82.15:4277 , piece_id=7NW6ZCVVR7ZR3PQ65CBIZENJJM3HRL3GMHGQOHL3UNYRXQP37BDA, seg_pos={0 0}, upload=2521]: Done uploading piece. err: ecclient: upload failed (node:1i493KQZuQYQwXzct9HamWDCq92CSGvLPHjZ6fxxVVZX1f7ofY, address:104.36.82.15:4277): protocol: write tcp 159.223.143.39:43806->104.36.82.15:4277: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:43.608831139 (2h44m59.389764524s) [noise=true , address=104.36.82.56:4252 , piece_id=R43YDGDQEUTT4PMDBBA6LHDXUZR4YCBKR43OCD3JEPQI65LQZTZQ, seg_pos={0 0}, upload=2521]: Done uploading piece. err: ecclient: upload failed (node:1UDFNiqXb4riRuewJDPrKFykVWLiryNKqgkH6uCZszz6Sr8naB, address:104.36.82.56:4252): protocol: write tcp 159.223.143.39:56482->104.36.82.56:4252: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:43.608878390 (2h44m59.389811753s) [noise=true , address=104.36.82.53:4151 , piece_id=OBTWN6PCKL4RUGRM4OBCLDLAAULA2RPDG73I2DHMLKQCXOSY3GCQ, seg_pos={0 0}, upload=2521]: Done uploading piece. err: ecclient: upload failed (node:196fMe3fbxKaPgNs1zQx7Kg3Rh68s6Gtp7mbu3hn654WCbZctx, address:104.36.82.53:4151): protocol: write tcp 159.223.143.39:52304->104.36.82.53:4151: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:43.608944475 (2h44m59.389877852s) [noise=true , address=104.36.82.44:4886 , piece_id=2AS3FEBOZ2EGVBG3EACVXFKOVE4MDY2CQI4HLPHAQWVMH35DHUPQ, seg_pos={0 0}, upload=2521]: Done uploading piece. err: ecclient: upload failed (node:1n2cXhqd5EWbNPTXem83jQUjYq5ZaGKBidbVCQzxYfR3rWddGR, address:104.36.82.44:4886): protocol: write tcp 159.223.143.39:43574->104.36.82.44:4886: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:43.608977323 (2h44m59.389910699s) [noise=true , address=104.36.82.54:4168 , piece_id=GHOKLEOAY3CXMFBWG4ZPJWWO7KHYHMXSDJ3HLB5D5VHMBVOOLI2Q, seg_pos={0 0}, upload=2521]: Done uploading piece. err: ecclient: upload failed (node:1XH68Yk8yh2Qa8vcPRYAuLHdGR7Jy4KtSBKYKgwAJpdDvVu7up, address:104.36.82.54:4168): protocol: rpc: dial tcp 104.36.82.54:4168: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:43.609024323 (2h44m59.38995769s) [noise=true , address=104.36.82.46:4502 , piece_id=Q647ENMCYF7F4JWE4YCRY3VGLXOEQ5ZWQJ5CH6EDEDM2KZ6DLD6Q, seg_pos={0 0}, upload=2521]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: write tcp 159.223.143.39:36674->104.36.82.46:4502: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:36:43.609061434 (2h44m59.38999482s) [seg_pos={0 0}, upload=2521]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:36:43.634116116 (2h44m59.415049486s) [upload=2521]: Done uploading segments... Jul 27 00:36:43.634148701 (2h44m59.415082073s) [upload=2521]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.030499246s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..............................................X.............] duration: 190.600788ms FAIL (2025-07-27 00:36:25 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 190.627062ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [X.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XXX] duration: 30.043907035s FAIL (2025-07-27 00:36:04 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.043938721s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X..] duration: 1.729456901s FAIL (2025-07-27 00:35:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ddress=77.90.31.15:4697 , piece_id=YWGMR6V3T7IWJ23KWMU4FTW7RGXM4YTEJMJHGXTAL2KF43OM33QQ, seg_pos={0 0}, upload=2499]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 159.223.143.39:56494->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:14.410656154 (2h42m30.19158952s) [seg_pos={0 0}, upload=2499]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:34:14.410757181 (2h42m30.191690548s) [upload=2499]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:34:14.410762832 (2h42m30.1916962s) [upload=2499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:34:14.484473815 (2h42m30.2654072s) [upload=2499]: Done uploading segments... Jul 27 00:34:14.484496345 (2h42m30.265429712s) [upload=2499]: Upload finished. err: <nil> Jul 27 00:34:56.461651021 (2h43m12.242584399s) [upload=2509]: Starting upload Jul 27 00:34:56.461766208 (2h43m12.242699585s) [upload=2509]: Done starting upload Jul 27 00:34:56.461865252 (2h43m12.242798622s) [upload=2509]: Uploading segments... Jul 27 00:34:56.462288861 (2h43m12.243222237s) [upload=2509]: Got next segment. Inline: false Jul 27 00:34:56.462365899 (2h43m12.243299274s) [upload=2509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:34:56.462372065 (2h43m12.243305433s) [upload=2509]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:34:56.493030998 (2h43m12.273964394s) [seg_pos={0 0}, upload=2509]: Begin upload segment... Jul 27 00:34:56.493085756 (2h43m12.274019121s) [seg_pos={0 0}, upload=2509]: Done begin upload segment. Jul 27 00:34:56.493115534 (2h43m12.274048922s) [noise=true , address=45.91.116.12:4225 , piece_id=DN332VW7K2CQUFEQQSR2YRHRXYGBIDQ6KMPQXOBVEFSJIAUEFZUQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.493655317 (2h43m12.274588706s) [noise=true , address=45.91.116.10:4083 , piece_id=XEESEWHRO7B2N336FHKQAH6LQWFQIMDUNEJ7RTXGBWS6ZH4MDINA, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.494066457 (2h43m12.274999844s) [noise=true , address=77.90.31.10:4518 , piece_id=RYADUWY55SZYWWICCDRHBQ5X3FOK5BGM4QQRWRMTBEW74ABO3HRQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.494435771 (2h43m12.27536916s) [noise=true , address=104.36.82.16:4294 , piece_id=MATRG2ZHPTWDKLIWG7BZK2EFMVPTS4TZVQ5QXOAUVGMU2ZBPMWJA, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.494813297 (2h43m12.275746684s) [noise=true , address=45.91.116.5:4299 , piece_id=EAVTRJ7M3553N6INCYC7DLRILAURDN4BTGYE7EUPC56GCRD3M34A, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.495197053 (2h43m12.276130441s) [noise=true , address=45.91.116.11:4117 , piece_id=LKC3U3BYOKLK5KNTZQGU4XG6KNWFEWWI3WBAS7VRRVPIWJWTRO5Q, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.495884173 (2h43m12.276817562s) [noise=true , address=45.91.116.9:4263 , piece_id=UJRTMIDHZO2TZUJWIJYG2HN6DGLYDTBKJRMWLKCZ4MU6LPKWHD4Q, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.496330821 (2h43m12.277264209s) [noise=true , address=77.90.31.19:4842 , piece_id=GINHDM6UYOG47J3YFE3I3C6GQIXAZF6Q3U7ED4HANIB4N3CADBMA, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.496706877 (2h43m12.277640279s) [noise=true , address=104.36.82.6:4875 , piece_id=YQHAOMEBUHGHCQUIPGEWFPTR27H4KYM4JQEUK6X2KN3VD7XBJJIQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.497118167 (2h43m12.27805155s) [noise=true , address=45.91.116.8:4048 , piece_id=VXZ2BYUKODUSVYQWXFGO7Q6KWSZ6QKTLKN7VU6Q4K3FBPOZOAPBA, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.497620808 (2h43m12.278554199s) [noise=true , address=77.90.31.18:4807 , piece_id=PLZIYYA5EUE3MYEGWLPMFYUIF4IXHTGQRFXNG6B2VK34CFQ4FPGQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.498135578 (2h43m12.279068966s) [noise=true , address=77.90.31.17:4771 , piece_id=DGSX2GUYVTJS3NRUJGQS2PXJQNX3JOWWP5RFUMPAVYZQSKPF4Y7A, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.498550870 (2h43m12.279484255s) [noise=true , address=45.91.116.13:4154 , piece_id=ZJTNYP3AMC5DOFFW7BJNBRDIVHRMBOOWL4PG4O5OZA3GKCJLCCLQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.498636383 (2h43m12.279569891s) [noise=true , address=104.36.82.11:4115 , piece_id=2NWQCYEEA5WHWGF6Y3ZQMUQ6X4IJX3B6NTSJQRDFVYYMT3SMREDQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.499249260 (2h43m12.280182647s) [noise=true , address=45.91.116.7:4011 , piece_id=S7AJXPSVZ2CG6FIBBAKTXMXVNOZEENR422NZFVCGKQXYJGM46GUQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.499853655 (2h43m12.280787038s) [noise=true , address=77.90.31.14:4661 , piece_id=6MCSA25ML5YSMSLRGIOXX4P7B7FYSKE24KXLOV24P2THBEER7J3Q, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.610589127 (2h43m12.39152249s) [upload=2509]: Next returned nil segment Jul 27 00:34:56.610624344 (2h43m12.391557712s) [upload=2509]: Waiting for error group managing segments... Jul 27 00:34:56.616563466 (2h43m12.397496841s) [noise=true , address=45.91.116.8:4048 , piece_id=VXZ2BYUKODUSVYQWXFGO7Q6KWSZ6QKTLKN7VU6Q4K3FBPOZOAPBA, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.616709756 (2h43m12.39764313s) [noise=true , address=45.91.116.7:4011 , piece_id=S7AJXPSVZ2CG6FIBBAKTXMXVNOZEENR422NZFVCGKQXYJGM46GUQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.617035973 (2h43m12.397969334s) [noise=true , address=45.91.116.13:4154 , piece_id=ZJTNYP3AMC5DOFFW7BJNBRDIVHRMBOOWL4PG4O5OZA3GKCJLCCLQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.617711891 (2h43m12.398645253s) [noise=true , address=77.90.31.14:4661 , piece_id=6MCSA25ML5YSMSLRGIOXX4P7B7FYSKE24KXLOV24P2THBEER7J3Q, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.618591960 (2h43m12.399525328s) [noise=true , address=45.91.116.10:4083 , piece_id=XEESEWHRO7B2N336FHKQAH6LQWFQIMDUNEJ7RTXGBWS6ZH4MDINA, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.618791840 (2h43m12.399725209s) [noise=true , address=45.91.116.12:4225 , piece_id=DN332VW7K2CQUFEQQSR2YRHRXYGBIDQ6KMPQXOBVEFSJIAUEFZUQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.619282261 (2h43m12.400215628s) [noise=true , address=45.91.116.5:4299 , piece_id=EAVTRJ7M3553N6INCYC7DLRILAURDN4BTGYE7EUPC56GCRD3M34A, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.619415952 (2h43m12.400349322s) [noise=true , address=45.91.116.9:4263 , piece_id=UJRTMIDHZO2TZUJWIJYG2HN6DGLYDTBKJRMWLKCZ4MU6LPKWHD4Q, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.619730046 (2h43m12.400663414s) [noise=true , address=45.91.116.11:4117 , piece_id=LKC3U3BYOKLK5KNTZQGU4XG6KNWFEWWI3WBAS7VRRVPIWJWTRO5Q, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.621838436 (2h43m12.4027718s) [noise=true , address=77.90.31.17:4771 , piece_id=DGSX2GUYVTJS3NRUJGQS2PXJQNX3JOWWP5RFUMPAVYZQSKPF4Y7A, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:57.976398485 (2h43m13.757331864s) [noise=true , address=77.90.31.19:4842 , piece_id=GINHDM6UYOG47J3YFE3I3C6GQIXAZF6Q3U7ED4HANIB4N3CADBMA, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:57.988879135 (2h43m13.769812498s) [noise=true , address=77.90.31.10:4518 , piece_id=RYADUWY55SZYWWICCDRHBQ5X3FOK5BGM4QQRWRMTBEW74ABO3HRQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:35:26.417197611 (2h43m42.198131094s) [noise=true , address=104.36.82.16:4294 , piece_id=MATRG2ZHPTWDKLIWG7BZK2EFMVPTS4TZVQ5QXOAUVGMU2ZBPMWJA, seg_pos={0 0}, upload=2509]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 159.223.143.39:39154->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:26.417330209 (2h43m42.198263586s) [noise=true , address=104.36.82.6:4875 , piece_id=YQHAOMEBUHGHCQUIPGEWFPTR27H4KYM4JQEUK6X2KN3VD7XBJJIQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:26.417437864 (2h43m42.198371226s) [noise=true , address=104.36.82.11:4115 , piece_id=2NWQCYEEA5WHWGF6Y3ZQMUQ6X4IJX3B6NTSJQRDFVYYMT3SMREDQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:44308->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:26.418006183 (2h43m42.198939566s) [noise=true , address=77.90.31.18:4807 , piece_id=PLZIYYA5EUE3MYEGWLPMFYUIF4IXHTGQRFXNG6B2VK34CFQ4FPGQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:26.418076821 (2h43m42.199010209s) [seg_pos={0 0}, upload=2509]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:35:26.443959392 (2h43m42.224892772s) [upload=2509]: Done uploading segments... Jul 27 00:35:26.443992565 (2h43m42.224925943s) [upload=2509]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.72949451s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...............................................X............] duration: 201.583594ms FAIL (2025-07-27 00:35:55 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 201.622819ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X......X.XX....X...X.................XX....X......XX.XXXX.] duration: 9.549374267s FAIL (2025-07-27 00:35:49 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.549469115s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.045309488s FAIL (2025-07-27 00:35:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== ece_id=AB4HZ6W2JXGTYSHZREH5Q52K3JA33WFA2WRFYD4YSSZ3WD2K6PSQ, seg_pos={0 0}, upload=2505]: Uploading piece... Jul 27 00:34:43.573968430 (2h42m59.354901813s) [noise=true , address=79.127.219.38:4015 , piece_id=EGJNHAQMP5376WJE5K4EWCIZ7V5EP6LQJQEJV3NBWFZVTAP66APA, seg_pos={0 0}, upload=2505]: Uploading piece... Jul 27 00:34:43.574337612 (2h42m59.355270982s) [seg_pos={0 0}, upload=2505]: Done begin upload segment. Jul 27 00:34:43.574373029 (2h42m59.355306413s) [noise=true , address=79.127.205.235:4665 , piece_id=2KDIRP5HXY6FY3JXN6WD2OZ2YDLPAV5OHMUYHA533ZA3BRA77YBQ, seg_pos={0 0}, upload=2505]: Uploading piece... Jul 27 00:34:43.575401969 (2h42m59.35633535s) [noise=true , address=77.90.31.13:4644 , piece_id=X4IE4S6QRV3FTVSL63334AQ3GXVHSEKBT64NUYIU5TCZZMBURPAA, seg_pos={0 0}, upload=2505]: Uploading piece... Jul 27 00:34:43.576262217 (2h42m59.357195606s) [noise=true , address=104.36.82.46:4506 , piece_id=UQ7VTND7Q2XLIYDNWNNL6NUSZ5YPWSOCP2N5BYXQO5IBW22YV6BA, seg_pos={0 0}, upload=2505]: Uploading piece... Jul 27 00:34:43.578664828 (2h42m59.359598213s) [noise=true , address=77.90.31.21:4909 , piece_id=NZHWCSCJAQQ7TK3SOIGROIBWGOLKXC2WYKOUSSIHRGE6VIVHNVIA, seg_pos={0 0}, upload=2505]: Uploading piece... Jul 27 00:34:43.580948745 (2h42m59.36188213s) [noise=true , address=104.36.82.44:4880 , piece_id=PYRMHZ2GN3PYV3IW7IF5NMKWGANMGA2H744ON6JET3EKHR3QCS7A, seg_pos={0 0}, upload=2505]: Uploading piece... Jul 27 00:34:43.582593508 (2h42m59.363526893s) [noise=true , address=77.90.31.16:4736 , piece_id=BSEVRGHJSACRL4SRRQ5I6B5QMDPHH4BTGLEH5VTQZA35WRIATAEQ, seg_pos={0 0}, upload=2505]: Uploading piece... Jul 27 00:34:43.584983920 (2h42m59.3659173s) [noise=true , address=104.36.82.11:4137 , piece_id=FPSYKUYPBX34HQE3YG2I3YVMC2BEYNH6KQCKFSXER4RQB4CHSDMA, seg_pos={0 0}, upload=2505]: Uploading piece... Jul 27 00:34:43.585753066 (2h42m59.366686455s) [noise=true , address=104.36.82.55:4228 , piece_id=FTPI4X2QLTJK556QKIACULN3EEGYF3VJFVKT6WNQ3PKNZJUSY4NA, seg_pos={0 0}, upload=2505]: Uploading piece... Jul 27 00:34:43.586230700 (2h42m59.367164087s) [noise=true , address=79.127.201.215:4343 , piece_id=XF4S3VIJBH7MPC4U2J44G626WXWBG76UETY3IKSYT4PFIGOHVROQ, seg_pos={0 0}, upload=2505]: Uploading piece... Jul 27 00:34:43.588904726 (2h42m59.369838116s) [noise=true , address=79.127.205.234:4613 , piece_id=LD3HK4BAQIVQWC6P2ZDHR6PJL426V7KO6SARTK7O46VZI7IINWSQ, seg_pos={0 0}, upload=2505]: Uploading piece... Jul 27 00:34:43.589477683 (2h42m59.370411079s) [noise=true , address=79.127.205.246:4258 , piece_id=IP4TKNFTSXXG2H775RUNTEI2GWXO4FWY625COLSEJKBEU7SWBWYA, seg_pos={0 0}, upload=2505]: Uploading piece... Jul 27 00:34:43.590317532 (2h42m59.371250922s) [noise=true , address=77.90.31.10:4531 , piece_id=RDWE6KZGFTIJIA6P7OJQRNGAN36M7FYVRCXBNUNBTHUZ23DLWBUQ, seg_pos={0 0}, upload=2505]: Uploading piece... Jul 27 00:34:43.590870071 (2h42m59.371803454s) [noise=true , address=104.36.82.53:4133 , piece_id=WF3OLGTAYY5OUJHRUGIUBQB6L2I3MIJWVYNU6NVZYNPOPKZTVH3Q, seg_pos={0 0}, upload=2505]: Uploading piece... Jul 27 00:34:43.592480729 (2h42m59.373414117s) [noise=true , address=79.127.205.245:4223 , piece_id=ZTMKXGX7OLCQMII5KKLHGI3M5AKM5BAJKILE55QNGN3HBXHUPSMA, seg_pos={0 0}, upload=2505]: Uploading piece... Jul 27 00:34:43.593465722 (2h42m59.37439911s) [noise=true , address=77.90.31.11:4580 , piece_id=P3SJBTFSQF4SZVI3SR5PJUH4U2E5LMNQKA2ZZLPKQ33V5764BLCQ, seg_pos={0 0}, upload=2505]: Uploading piece... Jul 27 00:34:43.594131698 (2h42m59.375065085s) [noise=true , address=104.36.82.21:4508 , piece_id=ETD54QH42FXID4SFO2AGB2BGRUWI7EW2W3CAYYIVOJ2H2RN4H32Q, seg_pos={0 0}, upload=2505]: Uploading piece... Jul 27 00:34:43.658773633 (2h42m59.439706999s) [upload=2505]: Next returned nil segment Jul 27 00:34:43.658798864 (2h42m59.439732244s) [upload=2505]: Waiting for error group managing segments... Jul 27 00:34:43.662060593 (2h42m59.442993962s) [noise=true , address=77.90.31.11:4580 , piece_id=P3SJBTFSQF4SZVI3SR5PJUH4U2E5LMNQKA2ZZLPKQ33V5764BLCQ, seg_pos={0 0}, upload=2505]: Done uploading piece. err: <nil> Jul 27 00:34:43.678141064 (2h42m59.459074443s) [noise=true , address=79.127.219.38:4015 , piece_id=EGJNHAQMP5376WJE5K4EWCIZ7V5EP6LQJQEJV3NBWFZVTAP66APA, seg_pos={0 0}, upload=2505]: Done uploading piece. err: <nil> Jul 27 00:34:43.867679682 (2h42m59.648613061s) [noise=true , address=77.90.31.16:4736 , piece_id=BSEVRGHJSACRL4SRRQ5I6B5QMDPHH4BTGLEH5VTQZA35WRIATAEQ, seg_pos={0 0}, upload=2505]: Done uploading piece. err: <nil> Jul 27 00:34:44.176047191 (2h42m59.956980557s) [noise=true , address=79.127.205.235:4665 , piece_id=2KDIRP5HXY6FY3JXN6WD2OZ2YDLPAV5OHMUYHA533ZA3BRA77YBQ, seg_pos={0 0}, upload=2505]: Done uploading piece. err: <nil> Jul 27 00:34:44.209176170 (2h42m59.990109548s) [noise=true , address=79.127.205.246:4258 , piece_id=IP4TKNFTSXXG2H775RUNTEI2GWXO4FWY625COLSEJKBEU7SWBWYA, seg_pos={0 0}, upload=2505]: Done uploading piece. err: <nil> Jul 27 00:34:44.209868361 (2h42m59.990801733s) [noise=true , address=79.127.205.245:4223 , piece_id=ZTMKXGX7OLCQMII5KKLHGI3M5AKM5BAJKILE55QNGN3HBXHUPSMA, seg_pos={0 0}, upload=2505]: Done uploading piece. err: <nil> Jul 27 00:34:44.222970450 (2h43m0.003903821s) [noise=true , address=79.127.205.234:4613 , piece_id=LD3HK4BAQIVQWC6P2ZDHR6PJL426V7KO6SARTK7O46VZI7IINWSQ, seg_pos={0 0}, upload=2505]: Done uploading piece. err: <nil> Jul 27 00:34:47.278114694 (2h43m3.059048073s) [noise=true , address=77.90.31.10:4531 , piece_id=RDWE6KZGFTIJIA6P7OJQRNGAN36M7FYVRCXBNUNBTHUZ23DLWBUQ, seg_pos={0 0}, upload=2505]: Done uploading piece. err: <nil> Jul 27 00:35:06.093251298 (2h43m21.87418466s) [noise=true , address=77.90.31.13:4644 , piece_id=X4IE4S6QRV3FTVSL63334AQ3GXVHSEKBT64NUYIU5TCZZMBURPAA, seg_pos={0 0}, upload=2505]: Done uploading piece. err: <nil> Jul 27 00:35:11.393416664 (2h43m27.174350044s) [noise=true , address=79.127.201.215:4343 , piece_id=XF4S3VIJBH7MPC4U2J44G626WXWBG76UETY3IKSYT4PFIGOHVROQ, seg_pos={0 0}, upload=2505]: Done uploading piece. err: <nil> Jul 27 00:35:43.484832220 (2h43m59.265765584s) [noise=true , address=104.36.82.21:4508 , piece_id=ETD54QH42FXID4SFO2AGB2BGRUWI7EW2W3CAYYIVOJ2H2RN4H32Q, seg_pos={0 0}, upload=2505]: Done uploading piece. err: ecclient: upload failed (node:1269NRnqpabU6H8q9vSCmE8yMzznD1L7dyZFmzCWhtzJaTjbXg3, address:104.36.82.21:4508): protocol: rpc: dial tcp 104.36.82.21:4508: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:43.484908314 (2h43m59.265841678s) [noise=true , address=104.36.82.53:4133 , piece_id=WF3OLGTAYY5OUJHRUGIUBQB6L2I3MIJWVYNU6NVZYNPOPKZTVH3Q, seg_pos={0 0}, upload=2505]: Done uploading piece. err: ecclient: upload failed (node:12usLZNRYviwS7dfk6soCgE7XP5LeyVcSUpjhWAk3stm61tjR7g, address:104.36.82.53:4133): protocol: rpc: dial tcp 104.36.82.53:4133: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:43.484975164 (2h43m59.265908545s) [noise=true , address=77.90.31.21:4909 , piece_id=NZHWCSCJAQQ7TK3SOIGROIBWGOLKXC2WYKOUSSIHRGE6VIVHNVIA, seg_pos={0 0}, upload=2505]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: rpc: dial tcp 77.90.31.21:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:43.485043903 (2h43m59.265977286s) [noise=true , address=104.36.82.11:4137 , piece_id=FPSYKUYPBX34HQE3YG2I3YVMC2BEYNH6KQCKFSXER4RQB4CHSDMA, seg_pos={0 0}, upload=2505]: Done uploading piece. err: ecclient: upload failed (node:12tkbodtFrNCuJbvZk8YnpSdJfVsKasNuKhg2wGD8rDkNPrHZ21, address:104.36.82.11:4137): protocol: write tcp 159.223.143.39:35872->104.36.82.11:4137: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:43.485105252 (2h43m59.266038633s) [noise=true , address=104.36.82.52:4110 , piece_id=AB4HZ6W2JXGTYSHZREH5Q52K3JA33WFA2WRFYD4YSSZ3WD2K6PSQ, seg_pos={0 0}, upload=2505]: Done uploading piece. err: ecclient: upload failed (node:1ftGsmDR7RLZnGZzHLnE7g4PSTkpBo4JLN5MBaXPjPciPETYui, address:104.36.82.52:4110): protocol: rpc: dial tcp 104.36.82.52:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:43.485118982 (2h43m59.26605236s) [noise=true , address=104.36.82.46:4506 , piece_id=UQ7VTND7Q2XLIYDNWNNL6NUSZ5YPWSOCP2N5BYXQO5IBW22YV6BA, seg_pos={0 0}, upload=2505]: Done uploading piece. err: ecclient: upload failed (node:12o9EEYka41UdTcVAmP7BJy8vrtQ2h4kvi1wGjcnedX4UKJspL9, address:104.36.82.46:4506): protocol: write tcp 159.223.143.39:33556->104.36.82.46:4506: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:43.485183669 (2h43m59.266117046s) [noise=true , address=104.36.82.44:4880 , piece_id=PYRMHZ2GN3PYV3IW7IF5NMKWGANMGA2H744ON6JET3EKHR3QCS7A, seg_pos={0 0}, upload=2505]: Done uploading piece. err: ecclient: upload failed (node:1sFbwzYfjUGQqrKJhgwDg5dvormM1Xu4yc97awR5kgZUAsKiEn, address:104.36.82.44:4880): protocol: rpc: dial tcp 104.36.82.44:4880: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:43.485190175 (2h43m59.266123551s) [noise=true , address=104.36.82.55:4228 , piece_id=FTPI4X2QLTJK556QKIACULN3EEGYF3VJFVKT6WNQ3PKNZJUSY4NA, seg_pos={0 0}, upload=2505]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: rpc: dial tcp 104.36.82.55:4228: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:43.485232676 (2h43m59.266166064s) [seg_pos={0 0}, upload=2505]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:35:43.523512888 (2h43m59.304446266s) [upload=2505]: Done uploading segments... Jul 27 00:35:43.523548101 (2h43m59.304481483s) [upload=2505]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.045337179s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [.......X...XX...............................X...............] duration: 24.731139256s FAIL (2025-07-27 00:35:39 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.731161916s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X.] duration: 499.003171ms FAIL (2025-07-27 00:35:27 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ddress=77.90.31.15:4697 , piece_id=YWGMR6V3T7IWJ23KWMU4FTW7RGXM4YTEJMJHGXTAL2KF43OM33QQ, seg_pos={0 0}, upload=2499]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 159.223.143.39:56494->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:14.410656154 (2h42m30.19158952s) [seg_pos={0 0}, upload=2499]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:34:14.410757181 (2h42m30.191690548s) [upload=2499]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:34:14.410762832 (2h42m30.1916962s) [upload=2499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:34:14.484473815 (2h42m30.2654072s) [upload=2499]: Done uploading segments... Jul 27 00:34:14.484496345 (2h42m30.265429712s) [upload=2499]: Upload finished. err: <nil> Jul 27 00:34:56.461651021 (2h43m12.242584399s) [upload=2509]: Starting upload Jul 27 00:34:56.461766208 (2h43m12.242699585s) [upload=2509]: Done starting upload Jul 27 00:34:56.461865252 (2h43m12.242798622s) [upload=2509]: Uploading segments... Jul 27 00:34:56.462288861 (2h43m12.243222237s) [upload=2509]: Got next segment. Inline: false Jul 27 00:34:56.462365899 (2h43m12.243299274s) [upload=2509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:34:56.462372065 (2h43m12.243305433s) [upload=2509]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:34:56.493030998 (2h43m12.273964394s) [seg_pos={0 0}, upload=2509]: Begin upload segment... Jul 27 00:34:56.493085756 (2h43m12.274019121s) [seg_pos={0 0}, upload=2509]: Done begin upload segment. Jul 27 00:34:56.493115534 (2h43m12.274048922s) [noise=true , address=45.91.116.12:4225 , piece_id=DN332VW7K2CQUFEQQSR2YRHRXYGBIDQ6KMPQXOBVEFSJIAUEFZUQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.493655317 (2h43m12.274588706s) [noise=true , address=45.91.116.10:4083 , piece_id=XEESEWHRO7B2N336FHKQAH6LQWFQIMDUNEJ7RTXGBWS6ZH4MDINA, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.494066457 (2h43m12.274999844s) [noise=true , address=77.90.31.10:4518 , piece_id=RYADUWY55SZYWWICCDRHBQ5X3FOK5BGM4QQRWRMTBEW74ABO3HRQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.494435771 (2h43m12.27536916s) [noise=true , address=104.36.82.16:4294 , piece_id=MATRG2ZHPTWDKLIWG7BZK2EFMVPTS4TZVQ5QXOAUVGMU2ZBPMWJA, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.494813297 (2h43m12.275746684s) [noise=true , address=45.91.116.5:4299 , piece_id=EAVTRJ7M3553N6INCYC7DLRILAURDN4BTGYE7EUPC56GCRD3M34A, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.495197053 (2h43m12.276130441s) [noise=true , address=45.91.116.11:4117 , piece_id=LKC3U3BYOKLK5KNTZQGU4XG6KNWFEWWI3WBAS7VRRVPIWJWTRO5Q, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.495884173 (2h43m12.276817562s) [noise=true , address=45.91.116.9:4263 , piece_id=UJRTMIDHZO2TZUJWIJYG2HN6DGLYDTBKJRMWLKCZ4MU6LPKWHD4Q, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.496330821 (2h43m12.277264209s) [noise=true , address=77.90.31.19:4842 , piece_id=GINHDM6UYOG47J3YFE3I3C6GQIXAZF6Q3U7ED4HANIB4N3CADBMA, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.496706877 (2h43m12.277640279s) [noise=true , address=104.36.82.6:4875 , piece_id=YQHAOMEBUHGHCQUIPGEWFPTR27H4KYM4JQEUK6X2KN3VD7XBJJIQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.497118167 (2h43m12.27805155s) [noise=true , address=45.91.116.8:4048 , piece_id=VXZ2BYUKODUSVYQWXFGO7Q6KWSZ6QKTLKN7VU6Q4K3FBPOZOAPBA, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.497620808 (2h43m12.278554199s) [noise=true , address=77.90.31.18:4807 , piece_id=PLZIYYA5EUE3MYEGWLPMFYUIF4IXHTGQRFXNG6B2VK34CFQ4FPGQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.498135578 (2h43m12.279068966s) [noise=true , address=77.90.31.17:4771 , piece_id=DGSX2GUYVTJS3NRUJGQS2PXJQNX3JOWWP5RFUMPAVYZQSKPF4Y7A, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.498550870 (2h43m12.279484255s) [noise=true , address=45.91.116.13:4154 , piece_id=ZJTNYP3AMC5DOFFW7BJNBRDIVHRMBOOWL4PG4O5OZA3GKCJLCCLQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.498636383 (2h43m12.279569891s) [noise=true , address=104.36.82.11:4115 , piece_id=2NWQCYEEA5WHWGF6Y3ZQMUQ6X4IJX3B6NTSJQRDFVYYMT3SMREDQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.499249260 (2h43m12.280182647s) [noise=true , address=45.91.116.7:4011 , piece_id=S7AJXPSVZ2CG6FIBBAKTXMXVNOZEENR422NZFVCGKQXYJGM46GUQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.499853655 (2h43m12.280787038s) [noise=true , address=77.90.31.14:4661 , piece_id=6MCSA25ML5YSMSLRGIOXX4P7B7FYSKE24KXLOV24P2THBEER7J3Q, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.610589127 (2h43m12.39152249s) [upload=2509]: Next returned nil segment Jul 27 00:34:56.610624344 (2h43m12.391557712s) [upload=2509]: Waiting for error group managing segments... Jul 27 00:34:56.616563466 (2h43m12.397496841s) [noise=true , address=45.91.116.8:4048 , piece_id=VXZ2BYUKODUSVYQWXFGO7Q6KWSZ6QKTLKN7VU6Q4K3FBPOZOAPBA, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.616709756 (2h43m12.39764313s) [noise=true , address=45.91.116.7:4011 , piece_id=S7AJXPSVZ2CG6FIBBAKTXMXVNOZEENR422NZFVCGKQXYJGM46GUQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.617035973 (2h43m12.397969334s) [noise=true , address=45.91.116.13:4154 , piece_id=ZJTNYP3AMC5DOFFW7BJNBRDIVHRMBOOWL4PG4O5OZA3GKCJLCCLQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.617711891 (2h43m12.398645253s) [noise=true , address=77.90.31.14:4661 , piece_id=6MCSA25ML5YSMSLRGIOXX4P7B7FYSKE24KXLOV24P2THBEER7J3Q, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.618591960 (2h43m12.399525328s) [noise=true , address=45.91.116.10:4083 , piece_id=XEESEWHRO7B2N336FHKQAH6LQWFQIMDUNEJ7RTXGBWS6ZH4MDINA, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.618791840 (2h43m12.399725209s) [noise=true , address=45.91.116.12:4225 , piece_id=DN332VW7K2CQUFEQQSR2YRHRXYGBIDQ6KMPQXOBVEFSJIAUEFZUQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.619282261 (2h43m12.400215628s) [noise=true , address=45.91.116.5:4299 , piece_id=EAVTRJ7M3553N6INCYC7DLRILAURDN4BTGYE7EUPC56GCRD3M34A, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.619415952 (2h43m12.400349322s) [noise=true , address=45.91.116.9:4263 , piece_id=UJRTMIDHZO2TZUJWIJYG2HN6DGLYDTBKJRMWLKCZ4MU6LPKWHD4Q, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.619730046 (2h43m12.400663414s) [noise=true , address=45.91.116.11:4117 , piece_id=LKC3U3BYOKLK5KNTZQGU4XG6KNWFEWWI3WBAS7VRRVPIWJWTRO5Q, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.621838436 (2h43m12.4027718s) [noise=true , address=77.90.31.17:4771 , piece_id=DGSX2GUYVTJS3NRUJGQS2PXJQNX3JOWWP5RFUMPAVYZQSKPF4Y7A, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:57.976398485 (2h43m13.757331864s) [noise=true , address=77.90.31.19:4842 , piece_id=GINHDM6UYOG47J3YFE3I3C6GQIXAZF6Q3U7ED4HANIB4N3CADBMA, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:57.988879135 (2h43m13.769812498s) [noise=true , address=77.90.31.10:4518 , piece_id=RYADUWY55SZYWWICCDRHBQ5X3FOK5BGM4QQRWRMTBEW74ABO3HRQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:35:26.417197611 (2h43m42.198131094s) [noise=true , address=104.36.82.16:4294 , piece_id=MATRG2ZHPTWDKLIWG7BZK2EFMVPTS4TZVQ5QXOAUVGMU2ZBPMWJA, seg_pos={0 0}, upload=2509]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 159.223.143.39:39154->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:26.417330209 (2h43m42.198263586s) [noise=true , address=104.36.82.6:4875 , piece_id=YQHAOMEBUHGHCQUIPGEWFPTR27H4KYM4JQEUK6X2KN3VD7XBJJIQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:26.417437864 (2h43m42.198371226s) [noise=true , address=104.36.82.11:4115 , piece_id=2NWQCYEEA5WHWGF6Y3ZQMUQ6X4IJX3B6NTSJQRDFVYYMT3SMREDQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:44308->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:26.418006183 (2h43m42.198939566s) [noise=true , address=77.90.31.18:4807 , piece_id=PLZIYYA5EUE3MYEGWLPMFYUIF4IXHTGQRFXNG6B2VK34CFQ4FPGQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:26.418076821 (2h43m42.199010209s) [seg_pos={0 0}, upload=2509]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:35:26.443959392 (2h43m42.224892772s) [upload=2509]: Done uploading segments... Jul 27 00:35:26.443992565 (2h43m42.224925943s) [upload=2509]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 499.034484ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.X] duration: 30.034186304s FAIL (2025-07-27 00:35:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== ddress=77.90.31.15:4697 , piece_id=YWGMR6V3T7IWJ23KWMU4FTW7RGXM4YTEJMJHGXTAL2KF43OM33QQ, seg_pos={0 0}, upload=2499]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5): protocol: write tcp 159.223.143.39:56494->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:14.410656154 (2h42m30.19158952s) [seg_pos={0 0}, upload=2499]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:34:14.410757181 (2h42m30.191690548s) [upload=2499]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:34:14.410762832 (2h42m30.1916962s) [upload=2499]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:34:14.484473815 (2h42m30.2654072s) [upload=2499]: Done uploading segments... Jul 27 00:34:14.484496345 (2h42m30.265429712s) [upload=2499]: Upload finished. err: <nil> Jul 27 00:34:56.461651021 (2h43m12.242584399s) [upload=2509]: Starting upload Jul 27 00:34:56.461766208 (2h43m12.242699585s) [upload=2509]: Done starting upload Jul 27 00:34:56.461865252 (2h43m12.242798622s) [upload=2509]: Uploading segments... Jul 27 00:34:56.462288861 (2h43m12.243222237s) [upload=2509]: Got next segment. Inline: false Jul 27 00:34:56.462365899 (2h43m12.243299274s) [upload=2509]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:34:56.462372065 (2h43m12.243305433s) [upload=2509]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:34:56.493030998 (2h43m12.273964394s) [seg_pos={0 0}, upload=2509]: Begin upload segment... Jul 27 00:34:56.493085756 (2h43m12.274019121s) [seg_pos={0 0}, upload=2509]: Done begin upload segment. Jul 27 00:34:56.493115534 (2h43m12.274048922s) [noise=true , address=45.91.116.12:4225 , piece_id=DN332VW7K2CQUFEQQSR2YRHRXYGBIDQ6KMPQXOBVEFSJIAUEFZUQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.493655317 (2h43m12.274588706s) [noise=true , address=45.91.116.10:4083 , piece_id=XEESEWHRO7B2N336FHKQAH6LQWFQIMDUNEJ7RTXGBWS6ZH4MDINA, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.494066457 (2h43m12.274999844s) [noise=true , address=77.90.31.10:4518 , piece_id=RYADUWY55SZYWWICCDRHBQ5X3FOK5BGM4QQRWRMTBEW74ABO3HRQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.494435771 (2h43m12.27536916s) [noise=true , address=104.36.82.16:4294 , piece_id=MATRG2ZHPTWDKLIWG7BZK2EFMVPTS4TZVQ5QXOAUVGMU2ZBPMWJA, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.494813297 (2h43m12.275746684s) [noise=true , address=45.91.116.5:4299 , piece_id=EAVTRJ7M3553N6INCYC7DLRILAURDN4BTGYE7EUPC56GCRD3M34A, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.495197053 (2h43m12.276130441s) [noise=true , address=45.91.116.11:4117 , piece_id=LKC3U3BYOKLK5KNTZQGU4XG6KNWFEWWI3WBAS7VRRVPIWJWTRO5Q, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.495884173 (2h43m12.276817562s) [noise=true , address=45.91.116.9:4263 , piece_id=UJRTMIDHZO2TZUJWIJYG2HN6DGLYDTBKJRMWLKCZ4MU6LPKWHD4Q, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.496330821 (2h43m12.277264209s) [noise=true , address=77.90.31.19:4842 , piece_id=GINHDM6UYOG47J3YFE3I3C6GQIXAZF6Q3U7ED4HANIB4N3CADBMA, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.496706877 (2h43m12.277640279s) [noise=true , address=104.36.82.6:4875 , piece_id=YQHAOMEBUHGHCQUIPGEWFPTR27H4KYM4JQEUK6X2KN3VD7XBJJIQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.497118167 (2h43m12.27805155s) [noise=true , address=45.91.116.8:4048 , piece_id=VXZ2BYUKODUSVYQWXFGO7Q6KWSZ6QKTLKN7VU6Q4K3FBPOZOAPBA, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.497620808 (2h43m12.278554199s) [noise=true , address=77.90.31.18:4807 , piece_id=PLZIYYA5EUE3MYEGWLPMFYUIF4IXHTGQRFXNG6B2VK34CFQ4FPGQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.498135578 (2h43m12.279068966s) [noise=true , address=77.90.31.17:4771 , piece_id=DGSX2GUYVTJS3NRUJGQS2PXJQNX3JOWWP5RFUMPAVYZQSKPF4Y7A, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.498550870 (2h43m12.279484255s) [noise=true , address=45.91.116.13:4154 , piece_id=ZJTNYP3AMC5DOFFW7BJNBRDIVHRMBOOWL4PG4O5OZA3GKCJLCCLQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.498636383 (2h43m12.279569891s) [noise=true , address=104.36.82.11:4115 , piece_id=2NWQCYEEA5WHWGF6Y3ZQMUQ6X4IJX3B6NTSJQRDFVYYMT3SMREDQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.499249260 (2h43m12.280182647s) [noise=true , address=45.91.116.7:4011 , piece_id=S7AJXPSVZ2CG6FIBBAKTXMXVNOZEENR422NZFVCGKQXYJGM46GUQ, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.499853655 (2h43m12.280787038s) [noise=true , address=77.90.31.14:4661 , piece_id=6MCSA25ML5YSMSLRGIOXX4P7B7FYSKE24KXLOV24P2THBEER7J3Q, seg_pos={0 0}, upload=2509]: Uploading piece... Jul 27 00:34:56.610589127 (2h43m12.39152249s) [upload=2509]: Next returned nil segment Jul 27 00:34:56.610624344 (2h43m12.391557712s) [upload=2509]: Waiting for error group managing segments... Jul 27 00:34:56.616563466 (2h43m12.397496841s) [noise=true , address=45.91.116.8:4048 , piece_id=VXZ2BYUKODUSVYQWXFGO7Q6KWSZ6QKTLKN7VU6Q4K3FBPOZOAPBA, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.616709756 (2h43m12.39764313s) [noise=true , address=45.91.116.7:4011 , piece_id=S7AJXPSVZ2CG6FIBBAKTXMXVNOZEENR422NZFVCGKQXYJGM46GUQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.617035973 (2h43m12.397969334s) [noise=true , address=45.91.116.13:4154 , piece_id=ZJTNYP3AMC5DOFFW7BJNBRDIVHRMBOOWL4PG4O5OZA3GKCJLCCLQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.617711891 (2h43m12.398645253s) [noise=true , address=77.90.31.14:4661 , piece_id=6MCSA25ML5YSMSLRGIOXX4P7B7FYSKE24KXLOV24P2THBEER7J3Q, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.618591960 (2h43m12.399525328s) [noise=true , address=45.91.116.10:4083 , piece_id=XEESEWHRO7B2N336FHKQAH6LQWFQIMDUNEJ7RTXGBWS6ZH4MDINA, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.618791840 (2h43m12.399725209s) [noise=true , address=45.91.116.12:4225 , piece_id=DN332VW7K2CQUFEQQSR2YRHRXYGBIDQ6KMPQXOBVEFSJIAUEFZUQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.619282261 (2h43m12.400215628s) [noise=true , address=45.91.116.5:4299 , piece_id=EAVTRJ7M3553N6INCYC7DLRILAURDN4BTGYE7EUPC56GCRD3M34A, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.619415952 (2h43m12.400349322s) [noise=true , address=45.91.116.9:4263 , piece_id=UJRTMIDHZO2TZUJWIJYG2HN6DGLYDTBKJRMWLKCZ4MU6LPKWHD4Q, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.619730046 (2h43m12.400663414s) [noise=true , address=45.91.116.11:4117 , piece_id=LKC3U3BYOKLK5KNTZQGU4XG6KNWFEWWI3WBAS7VRRVPIWJWTRO5Q, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:56.621838436 (2h43m12.4027718s) [noise=true , address=77.90.31.17:4771 , piece_id=DGSX2GUYVTJS3NRUJGQS2PXJQNX3JOWWP5RFUMPAVYZQSKPF4Y7A, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:57.976398485 (2h43m13.757331864s) [noise=true , address=77.90.31.19:4842 , piece_id=GINHDM6UYOG47J3YFE3I3C6GQIXAZF6Q3U7ED4HANIB4N3CADBMA, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:34:57.988879135 (2h43m13.769812498s) [noise=true , address=77.90.31.10:4518 , piece_id=RYADUWY55SZYWWICCDRHBQ5X3FOK5BGM4QQRWRMTBEW74ABO3HRQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: <nil> Jul 27 00:35:26.417197611 (2h43m42.198131094s) [noise=true , address=104.36.82.16:4294 , piece_id=MATRG2ZHPTWDKLIWG7BZK2EFMVPTS4TZVQ5QXOAUVGMU2ZBPMWJA, seg_pos={0 0}, upload=2509]: Done uploading piece. err: ecclient: upload failed (node:12CbfkWK4H84My3e6qhDuAkTrHWsooysz7KE4bk6VoDRDWEgb9e, address:104.36.82.16:4294): protocol: write tcp 159.223.143.39:39154->104.36.82.16:4294: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:26.417330209 (2h43m42.198263586s) [noise=true , address=104.36.82.6:4875 , piece_id=YQHAOMEBUHGHCQUIPGEWFPTR27H4KYM4JQEUK6X2KN3VD7XBJJIQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:26.417437864 (2h43m42.198371226s) [noise=true , address=104.36.82.11:4115 , piece_id=2NWQCYEEA5WHWGF6Y3ZQMUQ6X4IJX3B6NTSJQRDFVYYMT3SMREDQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:44308->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:26.418006183 (2h43m42.198939566s) [noise=true , address=77.90.31.18:4807 , piece_id=PLZIYYA5EUE3MYEGWLPMFYUIF4IXHTGQRFXNG6B2VK34CFQ4FPGQ, seg_pos={0 0}, upload=2509]: Done uploading piece. err: ecclient: upload failed (node:12jVDPs8JFrPp3swUR5wBoEHpii8BS9F5Hb5p3vUe6btGKHfYaS, address:77.90.31.18:4807): protocol: rpc: dial tcp 77.90.31.18:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:35:26.418076821 (2h43m42.199010209s) [seg_pos={0 0}, upload=2509]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:35:26.443959392 (2h43m42.224892772s) [upload=2509]: Done uploading segments... Jul 27 00:35:26.443992565 (2h43m42.224925943s) [upload=2509]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.0342234s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X..X......X.XX....X...X.................XX....X......XX.XXXX] duration: 30.000709457s FAIL (2025-07-27 00:35:24 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000736333s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [........X...XX...............................X..............] duration: 30.35119273s FAIL (2025-07-27 00:35:14 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.351230162s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [XX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.XX] duration: 30.043034358s FAIL (2025-07-27 00:35:09 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.043064479s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [................................................X...........] duration: 230.265671ms FAIL (2025-07-27 00:35:02 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 230.297944ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.03060561s FAIL (2025-07-27 00:34:43 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.487641602 (2h41m59.268574979s) [noise=true , address=77.90.31.13:4644 , piece_id=4L7FNU2PUPO7OYV3N6ZL5DWTL336XNDYHDYSCVEXFKKEWSJ4LL6A, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.487701637 (2h41m59.268635024s) [noise=true , address=104.36.82.55:4223 , piece_id=WD57UZBF4AKX7BFURA3ISAJZM6RUQ5IOFPCBZHF4ZA7IAHLJP3GA, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.488179834 (2h41m59.269113213s) [noise=true , address=79.127.205.235:4657 , piece_id=CGOAGQQTZZHY7IY52WDKKT55ILUXKUDB7RTGLIWCG6MC4CEAJGOA, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.488384882 (2h41m59.269318269s) [noise=true , address=77.90.31.15:4714 , piece_id=XXQD63XWG5KMB7TIL2PP77C4AWBNARGNEWQ3EUTGFRJC4UEW6DRA, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.488882777 (2h41m59.269816172s) [noise=true , address=77.90.31.21:4902 , piece_id=HKRSE6BQMT3YIPQ62HWY3NAUPXL56TO2Y4ZQRLTB6T3RWFF7QDCQ, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.489467222 (2h41m59.270400614s) [noise=true , address=77.90.31.11:4579 , piece_id=EM637FGIU7ZXWCUJMTGGH5RLGORFVVGFEOYW2SKUXB7G4W6PFOFQ, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.490024741 (2h41m59.270958126s) [noise=true , address=104.36.82.46:4508 , piece_id=JWDLNFNJK2552QNUPJ5QHR5EZJOCJCVXY5U6WZRXNPRFNQOEEZGQ, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.490561936 (2h41m59.271495323s) [noise=true , address=104.36.82.47:4547 , piece_id=XIXPY7DNB42KIZT5IFF4NN5XKX6EMGMTJLVYLLXHL6QPIVJJXCJQ, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.491132129 (2h41m59.272065511s) [noise=true , address=156.146.43.225:4448 , piece_id=VQJQIOUKHAAGGPPKWPNB36V6YB2DTZ3CG26AVZHYURQDWWKSTWMQ, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.491664941 (2h41m59.272598325s) [noise=true , address=104.36.82.52:4096 , piece_id=HNI4KQC3RTYTOK6XVOJ2DYSQ7O3AGBS7B7BPNTWL4XOEYEEVTPBA, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.492216720 (2h41m59.273150104s) [noise=true , address=64.49.13.138:4483 , piece_id=S573IWHSBPEMN637GQNPFVVXCDMWRBDLO2WNPVTOG4RK6BB2ISPA, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.492774668 (2h41m59.273708097s) [noise=true , address=104.36.82.54:4187 , piece_id=DP7A5VUNF5DAPEPBX7KKQXZBSL7WYTOZKUEYTM3JLYHWOMLS4EOQ, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.493402827 (2h41m59.274336218s) [noise=true , address=79.127.201.215:4343 , piece_id=TBRQWWYUDP7SOVEEUFGTJFBNTYNP2KEJUORZHMDAZSDMDOQIWHDQ, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.493949554 (2h41m59.274882945s) [noise=true , address=104.36.82.2:4748 , piece_id=TG2NMK4G4DBH2ONIPCRAMLV4GQGHKLGGJ5KGKRR5Q3HRZ5Y4EGKA, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.494489041 (2h41m59.27542246s) [noise=true , address=104.36.82.11:4127 , piece_id=5EV5RW3CTGV22KLLLAE7P5GIFAE76L7S6U6DCJIV6R7R6M3CGHCA, seg_pos={0 0}, upload=2490]: Uploading piece... Jul 27 00:33:43.597723882 (2h41m59.378657289s) [upload=2490]: Next returned nil segment Jul 27 00:33:43.597753328 (2h41m59.378686715s) [upload=2490]: Waiting for error group managing segments... Jul 27 00:33:43.614704527 (2h41m59.395637915s) [noise=true , address=77.90.31.11:4579 , piece_id=EM637FGIU7ZXWCUJMTGGH5RLGORFVVGFEOYW2SKUXB7G4W6PFOFQ, seg_pos={0 0}, upload=2490]: Done uploading piece. err: <nil> Jul 27 00:33:43.666532633 (2h41m59.447466003s) [noise=true , address=156.146.43.225:4448 , piece_id=VQJQIOUKHAAGGPPKWPNB36V6YB2DTZ3CG26AVZHYURQDWWKSTWMQ, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:1irai8UTYknGTUkKHtKbBNTJTYMQsNTFnzrTEaWCMRDq1NNpGh, address:156.146.43.225:4448): protocol: storage node overloaded, request limit: 2000 Jul 27 00:33:44.062967841 (2h41m59.843901221s) [noise=true , address=79.127.205.234:4610 , piece_id=IGVKOIZKCF33LO4KS5EWRDJLFRAFSXNVBVDO2D6AU3OQXJVGJG6A, seg_pos={0 0}, upload=2490]: Done uploading piece. err: <nil> Jul 27 00:33:44.075951518 (2h41m59.856884908s) [noise=true , address=79.127.219.38:4001 , piece_id=MJELZ7SSO24CWDJPDHCWTW4RSLGJ6EHGKHVGLJFG55FLAQXDYXWQ, seg_pos={0 0}, upload=2490]: Done uploading piece. err: <nil> Jul 27 00:33:44.130316939 (2h41m59.911250308s) [noise=true , address=79.127.205.235:4657 , piece_id=CGOAGQQTZZHY7IY52WDKKT55ILUXKUDB7RTGLIWCG6MC4CEAJGOA, seg_pos={0 0}, upload=2490]: Done uploading piece. err: <nil> Jul 27 00:33:45.126339405 (2h42m0.907272785s) [noise=true , address=64.49.13.138:4482 , piece_id=LAQISUWZNJQT4BB5AEVKBIS3S3CC2G7KN3FFPXSCHDSENCUHF7OQ, seg_pos={0 0}, upload=2490]: Done uploading piece. err: <nil> Jul 27 00:33:45.248105911 (2h42m1.029039298s) [noise=true , address=64.49.13.138:4483 , piece_id=S573IWHSBPEMN637GQNPFVVXCDMWRBDLO2WNPVTOG4RK6BB2ISPA, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 27 00:33:46.857255695 (2h42m2.638189063s) [noise=true , address=77.90.31.13:4644 , piece_id=4L7FNU2PUPO7OYV3N6ZL5DWTL336XNDYHDYSCVEXFKKEWSJ4LL6A, seg_pos={0 0}, upload=2490]: Done uploading piece. err: <nil> Jul 27 00:34:06.788607155 (2h42m22.569540535s) [noise=true , address=79.127.201.215:4343 , piece_id=TBRQWWYUDP7SOVEEUFGTJFBNTYNP2KEJUORZHMDAZSDMDOQIWHDQ, seg_pos={0 0}, upload=2490]: Done uploading piece. err: <nil> Jul 27 00:34:43.384486319 (2h42m59.165419701s) [noise=true , address=77.90.31.21:4902 , piece_id=HKRSE6BQMT3YIPQ62HWY3NAUPXL56TO2Y4ZQRLTB6T3RWFF7QDCQ, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:12sD3HFEaYLm7KV6uRvjbojtU9eQGMSy3Aewr4pX5Bxjy4jMCHx, address:77.90.31.21:4902): protocol: rpc: dial tcp 77.90.31.21:4902: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:43.384535758 (2h42m59.165469125s) [noise=true , address=104.36.82.2:4748 , piece_id=TG2NMK4G4DBH2ONIPCRAMLV4GQGHKLGGJ5KGKRR5Q3HRZ5Y4EGKA, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:125iuuffKDUuPufTd8CEM8ZzHodp4h2ZAhwegCq8sZPuNi4oLqv, address:104.36.82.2:4748): protocol: write tcp 159.223.143.39:43994->104.36.82.2:4748: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:43.384561873 (2h42m59.165495248s) [noise=true , address=104.36.82.52:4096 , piece_id=HNI4KQC3RTYTOK6XVOJ2DYSQ7O3AGBS7B7BPNTWL4XOEYEEVTPBA, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:1izNXqYvnRLt96tcnR1TXt29qcxZWEgACmHuJ6CikSgLbAzxLL, address:104.36.82.52:4096): protocol: rpc: dial tcp 104.36.82.52:4096: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:43.384630926 (2h42m59.165564288s) [noise=true , address=104.36.82.54:4187 , piece_id=DP7A5VUNF5DAPEPBX7KKQXZBSL7WYTOZKUEYTM3JLYHWOMLS4EOQ, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:12J7goNH48bnMJyaCAPgUznrX1n6bNFNZXGqpuftNLVLxQxprsF, address:104.36.82.54:4187): protocol: write tcp 159.223.143.39:34234->104.36.82.54:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:43.384638968 (2h42m59.165572332s) [noise=true , address=77.90.31.15:4714 , piece_id=XXQD63XWG5KMB7TIL2PP77C4AWBNARGNEWQ3EUTGFRJC4UEW6DRA, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, address:77.90.31.15:4714): protocol: rpc: dial tcp 77.90.31.15:4714: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:43.384720236 (2h42m59.165653614s) [noise=true , address=104.36.82.46:4508 , piece_id=JWDLNFNJK2552QNUPJ5QHR5EZJOCJCVXY5U6WZRXNPRFNQOEEZGQ, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:1zsSDfNdPmFgLDcojSxQio6MA9vh171m6hQbVEE7ZP9DRBC3vr, address:104.36.82.46:4508): protocol: rpc: dial tcp 104.36.82.46:4508: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:43.384728709 (2h42m59.165662074s) [noise=true , address=104.36.82.47:4547 , piece_id=XIXPY7DNB42KIZT5IFF4NN5XKX6EMGMTJLVYLLXHL6QPIVJJXCJQ, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:12brQg9WQMKJgp1buz1wMS8W11XEwAx7RHG5h51EmLvwY5D5Hky, address:104.36.82.47:4547): protocol: rpc: dial tcp 104.36.82.47:4547: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:43.384785454 (2h42m59.165718819s) [noise=true , address=104.36.82.55:4223 , piece_id=WD57UZBF4AKX7BFURA3ISAJZM6RUQ5IOFPCBZHF4ZA7IAHLJP3GA, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:1FijvRUiKwjk3A6yFZxNeLhPQ4bj5cxeEo6mjEs2DtRnK5WJr6, address:104.36.82.55:4223): protocol: rpc: dial tcp 104.36.82.55:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:43.384803466 (2h42m59.165736852s) [noise=true , address=104.36.82.11:4127 , piece_id=5EV5RW3CTGV22KLLLAE7P5GIFAE76L7S6U6DCJIV6R7R6M3CGHCA, seg_pos={0 0}, upload=2490]: Done uploading piece. err: ecclient: upload failed (node:12Pxos2xcjHwcYyQnXeRL3PL72G9pXaAnBs8XnkVFkxegDRPrqp, address:104.36.82.11:4127): protocol: write tcp 159.223.143.39:47242->104.36.82.11:4127: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:43.384845238 (2h42m59.165778624s) [seg_pos={0 0}, upload=2490]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 27 00:34:43.411651581 (2h42m59.192584981s) [upload=2490]: Done uploading segments... Jul 27 00:34:43.411684123 (2h42m59.192617516s) [upload=2490]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.030633949s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [X........X...XX...............................X.............] duration: 38.311359952s FAIL (2025-07-27 00:34:37 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 38.311387923s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X..X......X.XX....X...X.................XX....X......XX.XXX] duration: 30.000667811s FAIL (2025-07-27 00:34:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000685187s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX.] duration: 1.878007842s FAIL (2025-07-27 00:34:14 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nnection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:28.780254899 (2h41m44.561188289s) [seg_pos={0 0}, upload=2481]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:33:28.813631340 (2h41m44.594564717s) [upload=2481]: Done uploading segments... Jul 27 00:33:28.813655080 (2h41m44.594588456s) [upload=2481]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:33:42.772431891 (2h41m58.55336527s) [upload=2489]: Starting upload Jul 27 00:33:42.772531982 (2h41m58.553465363s) [upload=2489]: Done starting upload Jul 27 00:33:42.772598210 (2h41m58.553531573s) [upload=2489]: Uploading segments... Jul 27 00:33:42.772897554 (2h41m58.553830942s) [upload=2489]: Got next segment. Inline: false Jul 27 00:33:42.772966928 (2h41m58.55390031s) [upload=2489]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:33:42.772973908 (2h41m58.553907289s) [upload=2489]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:33:42.809452437 (2h41m58.590385807s) [seg_pos={0 0}, upload=2489]: Begin upload segment... Jul 27 00:33:42.809511211 (2h41m58.590444574s) [seg_pos={0 0}, upload=2489]: Done begin upload segment. Jul 27 00:33:42.809548227 (2h41m58.590481613s) [noise=true , address=104.36.82.14:4223 , piece_id=S5EWQ2GVNMNPUM2QA72JM3Y55KVRLPX7XIGKMXJMTB2HUHHBAY4A, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.810515534 (2h41m58.591448926s) [noise=true , address=45.91.116.10:4083 , piece_id=2A3PCZSWH6MTCNGZSR3ZEMQPYLQ6S2O4DHBBOPKELCIVVTMFXGDA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.810908555 (2h41m58.591841935s) [noise=true , address=45.91.116.13:4154 , piece_id=PHANYSQICLI44COK56IY5MMOQKSZQBG2J3OOJV73CVD66SD3BFWA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.813631673 (2h41m58.594565056s) [noise=true , address=104.36.82.10:4078 , piece_id=Q3UJWKRHCJEFIAHX7PTAHCKID3OSZIN5JAUOCOJGIRZ7AGQMHWEA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.814816475 (2h41m58.595749864s) [noise=true , address=45.91.116.4:4940 , piece_id=YUG6VMXYZDGH3IABFW4XF43TU4ZQGSNITXDOCIIDUYQ4ZZXIYC7A, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.815421088 (2h41m58.596354466s) [noise=true , address=104.36.82.4:4804 , piece_id=KZBEMTFRGP7LJKTCF5ZVFBJH5EFFGQIMNTICJ7U3FEHOPRALWKXQ, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.816024223 (2h41m58.59695761s) [noise=true , address=45.91.116.12:4226 , piece_id=FMD6MUY7UVHCBIYFD2BJX7WKJOGKCQUX56OT6ENPTDTYUUC2BYJA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.816505852 (2h41m58.597439232s) [noise=true , address=77.90.31.16:4735 , piece_id=X4EK2C2UXB74VDQHCUYMVZ2TGHWBT4XIKHZH53XEANZLYNDIZ7JA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.816981623 (2h41m58.597915006s) [noise=true , address=77.90.31.14:4662 , piece_id=UQFCPUYMOPYTHB7JMKJMD4XHN6GAQNJTA6DNJEQBPEGKWBSIJEFQ, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.817406255 (2h41m58.598339641s) [noise=true , address=45.91.116.11:4119 , piece_id=UX4UFKU5UFJVBF575IAWUJDPTUDANNDCAZA5BZLTGDKGEJK3TD7Q, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.817912249 (2h41m58.598845634s) [noise=true , address=45.91.116.6:4973 , piece_id=UZDWLZZVEB7G7CQXLQDFQBWLDFEPJP47FSQXQT4QI6RCWHOX37PA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.818405766 (2h41m58.599339154s) [noise=true , address=104.36.82.13:4187 , piece_id=VOUFP55K63GOPO7352M43MOF6OOZAMGGNNFWOZBPNTIYMYIP47XA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.818892203 (2h41m58.599825593s) [noise=true , address=45.91.116.14:4191 , piece_id=UVT23XLMFU7O6JVGPNGLGEYJBA27LCGESLADL3SOQNRFBZYFAPNA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.819335313 (2h41m58.600268697s) [noise=true , address=45.91.116.6:4974 , piece_id=QKXO7XNRATDMWCNIVSP6E6KWD4QBXK56R4HEPCEWAE6TIS4GLFQA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.819881361 (2h41m58.60081475s) [noise=true , address=104.36.82.3:4767 , piece_id=O2WDAQJY4RBQVZ72EVWVBYZX44F7TLDMTL6GYCN3XG6VF2ILA67Q, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.820392722 (2h41m58.601326112s) [noise=true , address=104.36.82.9:4007 , piece_id=7D6VCXWVHACS3MKG2VT6RFPPDLKSMWQEZEDOS6FJ4A44YXULY5NA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.971323604 (2h41m58.752256991s) [upload=2489]: Next returned nil segment Jul 27 00:33:42.971346037 (2h41m58.75227942s) [upload=2489]: Waiting for error group managing segments... Jul 27 00:33:42.976454805 (2h41m58.757388175s) [noise=true , address=45.91.116.6:4974 , piece_id=QKXO7XNRATDMWCNIVSP6E6KWD4QBXK56R4HEPCEWAE6TIS4GLFQA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.979574580 (2h41m58.76050795s) [noise=true , address=45.91.116.10:4083 , piece_id=2A3PCZSWH6MTCNGZSR3ZEMQPYLQ6S2O4DHBBOPKELCIVVTMFXGDA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.979620139 (2h41m58.760553522s) [noise=true , address=45.91.116.12:4226 , piece_id=FMD6MUY7UVHCBIYFD2BJX7WKJOGKCQUX56OT6ENPTDTYUUC2BYJA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.980316822 (2h41m58.761250191s) [noise=true , address=45.91.116.6:4973 , piece_id=UZDWLZZVEB7G7CQXLQDFQBWLDFEPJP47FSQXQT4QI6RCWHOX37PA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.980396243 (2h41m58.761329628s) [noise=true , address=45.91.116.13:4154 , piece_id=PHANYSQICLI44COK56IY5MMOQKSZQBG2J3OOJV73CVD66SD3BFWA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.981311003 (2h41m58.762244372s) [noise=true , address=45.91.116.11:4119 , piece_id=UX4UFKU5UFJVBF575IAWUJDPTUDANNDCAZA5BZLTGDKGEJK3TD7Q, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.981996571 (2h41m58.762929941s) [noise=true , address=45.91.116.4:4940 , piece_id=YUG6VMXYZDGH3IABFW4XF43TU4ZQGSNITXDOCIIDUYQ4ZZXIYC7A, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.982071090 (2h41m58.763004455s) [noise=true , address=77.90.31.16:4735 , piece_id=X4EK2C2UXB74VDQHCUYMVZ2TGHWBT4XIKHZH53XEANZLYNDIZ7JA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.982578044 (2h41m58.763511411s) [noise=true , address=77.90.31.14:4662 , piece_id=UQFCPUYMOPYTHB7JMKJMD4XHN6GAQNJTA6DNJEQBPEGKWBSIJEFQ, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:43.139592182 (2h41m58.920525563s) [noise=true , address=45.91.116.14:4191 , piece_id=UVT23XLMFU7O6JVGPNGLGEYJBA27LCGESLADL3SOQNRFBZYFAPNA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:34:12.742401054 (2h42m28.523334439s) [noise=true , address=104.36.82.3:4767 , piece_id=O2WDAQJY4RBQVZ72EVWVBYZX44F7TLDMTL6GYCN3XG6VF2ILA67Q, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742488371 (2h42m28.523421747s) [noise=true , address=104.36.82.9:4007 , piece_id=7D6VCXWVHACS3MKG2VT6RFPPDLKSMWQEZEDOS6FJ4A44YXULY5NA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742542194 (2h42m28.523475563s) [noise=true , address=104.36.82.13:4187 , piece_id=VOUFP55K63GOPO7352M43MOF6OOZAMGGNNFWOZBPNTIYMYIP47XA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 159.223.143.39:43136->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742614945 (2h42m28.523548309s) [noise=true , address=104.36.82.14:4223 , piece_id=S5EWQ2GVNMNPUM2QA72JM3Y55KVRLPX7XIGKMXJMTB2HUHHBAY4A, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:41060->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742668062 (2h42m28.523601432s) [noise=true , address=104.36.82.4:4804 , piece_id=KZBEMTFRGP7LJKTCF5ZVFBJH5EFFGQIMNTICJ7U3FEHOPRALWKXQ, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:46106->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742752541 (2h42m28.523685906s) [noise=true , address=104.36.82.10:4078 , piece_id=Q3UJWKRHCJEFIAHX7PTAHCKID3OSZIN5JAUOCOJGIRZ7AGQMHWEA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:59466->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742794169 (2h42m28.523727553s) [seg_pos={0 0}, upload=2489]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:34:12.770005947 (2h42m28.550939315s) [upload=2489]: Done uploading segments... Jul 27 00:34:12.770043270 (2h42m28.550976637s) [upload=2489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.878035591s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.XX] duration: 30.029444024s FAIL (2025-07-27 00:34:12 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== nnection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:28.780254899 (2h41m44.561188289s) [seg_pos={0 0}, upload=2481]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:33:28.813631340 (2h41m44.594564717s) [upload=2481]: Done uploading segments... Jul 27 00:33:28.813655080 (2h41m44.594588456s) [upload=2481]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:33:42.772431891 (2h41m58.55336527s) [upload=2489]: Starting upload Jul 27 00:33:42.772531982 (2h41m58.553465363s) [upload=2489]: Done starting upload Jul 27 00:33:42.772598210 (2h41m58.553531573s) [upload=2489]: Uploading segments... Jul 27 00:33:42.772897554 (2h41m58.553830942s) [upload=2489]: Got next segment. Inline: false Jul 27 00:33:42.772966928 (2h41m58.55390031s) [upload=2489]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:33:42.772973908 (2h41m58.553907289s) [upload=2489]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:33:42.809452437 (2h41m58.590385807s) [seg_pos={0 0}, upload=2489]: Begin upload segment... Jul 27 00:33:42.809511211 (2h41m58.590444574s) [seg_pos={0 0}, upload=2489]: Done begin upload segment. Jul 27 00:33:42.809548227 (2h41m58.590481613s) [noise=true , address=104.36.82.14:4223 , piece_id=S5EWQ2GVNMNPUM2QA72JM3Y55KVRLPX7XIGKMXJMTB2HUHHBAY4A, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.810515534 (2h41m58.591448926s) [noise=true , address=45.91.116.10:4083 , piece_id=2A3PCZSWH6MTCNGZSR3ZEMQPYLQ6S2O4DHBBOPKELCIVVTMFXGDA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.810908555 (2h41m58.591841935s) [noise=true , address=45.91.116.13:4154 , piece_id=PHANYSQICLI44COK56IY5MMOQKSZQBG2J3OOJV73CVD66SD3BFWA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.813631673 (2h41m58.594565056s) [noise=true , address=104.36.82.10:4078 , piece_id=Q3UJWKRHCJEFIAHX7PTAHCKID3OSZIN5JAUOCOJGIRZ7AGQMHWEA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.814816475 (2h41m58.595749864s) [noise=true , address=45.91.116.4:4940 , piece_id=YUG6VMXYZDGH3IABFW4XF43TU4ZQGSNITXDOCIIDUYQ4ZZXIYC7A, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.815421088 (2h41m58.596354466s) [noise=true , address=104.36.82.4:4804 , piece_id=KZBEMTFRGP7LJKTCF5ZVFBJH5EFFGQIMNTICJ7U3FEHOPRALWKXQ, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.816024223 (2h41m58.59695761s) [noise=true , address=45.91.116.12:4226 , piece_id=FMD6MUY7UVHCBIYFD2BJX7WKJOGKCQUX56OT6ENPTDTYUUC2BYJA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.816505852 (2h41m58.597439232s) [noise=true , address=77.90.31.16:4735 , piece_id=X4EK2C2UXB74VDQHCUYMVZ2TGHWBT4XIKHZH53XEANZLYNDIZ7JA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.816981623 (2h41m58.597915006s) [noise=true , address=77.90.31.14:4662 , piece_id=UQFCPUYMOPYTHB7JMKJMD4XHN6GAQNJTA6DNJEQBPEGKWBSIJEFQ, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.817406255 (2h41m58.598339641s) [noise=true , address=45.91.116.11:4119 , piece_id=UX4UFKU5UFJVBF575IAWUJDPTUDANNDCAZA5BZLTGDKGEJK3TD7Q, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.817912249 (2h41m58.598845634s) [noise=true , address=45.91.116.6:4973 , piece_id=UZDWLZZVEB7G7CQXLQDFQBWLDFEPJP47FSQXQT4QI6RCWHOX37PA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.818405766 (2h41m58.599339154s) [noise=true , address=104.36.82.13:4187 , piece_id=VOUFP55K63GOPO7352M43MOF6OOZAMGGNNFWOZBPNTIYMYIP47XA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.818892203 (2h41m58.599825593s) [noise=true , address=45.91.116.14:4191 , piece_id=UVT23XLMFU7O6JVGPNGLGEYJBA27LCGESLADL3SOQNRFBZYFAPNA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.819335313 (2h41m58.600268697s) [noise=true , address=45.91.116.6:4974 , piece_id=QKXO7XNRATDMWCNIVSP6E6KWD4QBXK56R4HEPCEWAE6TIS4GLFQA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.819881361 (2h41m58.60081475s) [noise=true , address=104.36.82.3:4767 , piece_id=O2WDAQJY4RBQVZ72EVWVBYZX44F7TLDMTL6GYCN3XG6VF2ILA67Q, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.820392722 (2h41m58.601326112s) [noise=true , address=104.36.82.9:4007 , piece_id=7D6VCXWVHACS3MKG2VT6RFPPDLKSMWQEZEDOS6FJ4A44YXULY5NA, seg_pos={0 0}, upload=2489]: Uploading piece... Jul 27 00:33:42.971323604 (2h41m58.752256991s) [upload=2489]: Next returned nil segment Jul 27 00:33:42.971346037 (2h41m58.75227942s) [upload=2489]: Waiting for error group managing segments... Jul 27 00:33:42.976454805 (2h41m58.757388175s) [noise=true , address=45.91.116.6:4974 , piece_id=QKXO7XNRATDMWCNIVSP6E6KWD4QBXK56R4HEPCEWAE6TIS4GLFQA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.979574580 (2h41m58.76050795s) [noise=true , address=45.91.116.10:4083 , piece_id=2A3PCZSWH6MTCNGZSR3ZEMQPYLQ6S2O4DHBBOPKELCIVVTMFXGDA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.979620139 (2h41m58.760553522s) [noise=true , address=45.91.116.12:4226 , piece_id=FMD6MUY7UVHCBIYFD2BJX7WKJOGKCQUX56OT6ENPTDTYUUC2BYJA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.980316822 (2h41m58.761250191s) [noise=true , address=45.91.116.6:4973 , piece_id=UZDWLZZVEB7G7CQXLQDFQBWLDFEPJP47FSQXQT4QI6RCWHOX37PA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.980396243 (2h41m58.761329628s) [noise=true , address=45.91.116.13:4154 , piece_id=PHANYSQICLI44COK56IY5MMOQKSZQBG2J3OOJV73CVD66SD3BFWA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.981311003 (2h41m58.762244372s) [noise=true , address=45.91.116.11:4119 , piece_id=UX4UFKU5UFJVBF575IAWUJDPTUDANNDCAZA5BZLTGDKGEJK3TD7Q, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.981996571 (2h41m58.762929941s) [noise=true , address=45.91.116.4:4940 , piece_id=YUG6VMXYZDGH3IABFW4XF43TU4ZQGSNITXDOCIIDUYQ4ZZXIYC7A, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.982071090 (2h41m58.763004455s) [noise=true , address=77.90.31.16:4735 , piece_id=X4EK2C2UXB74VDQHCUYMVZ2TGHWBT4XIKHZH53XEANZLYNDIZ7JA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:42.982578044 (2h41m58.763511411s) [noise=true , address=77.90.31.14:4662 , piece_id=UQFCPUYMOPYTHB7JMKJMD4XHN6GAQNJTA6DNJEQBPEGKWBSIJEFQ, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:33:43.139592182 (2h41m58.920525563s) [noise=true , address=45.91.116.14:4191 , piece_id=UVT23XLMFU7O6JVGPNGLGEYJBA27LCGESLADL3SOQNRFBZYFAPNA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: <nil> Jul 27 00:34:12.742401054 (2h42m28.523334439s) [noise=true , address=104.36.82.3:4767 , piece_id=O2WDAQJY4RBQVZ72EVWVBYZX44F7TLDMTL6GYCN3XG6VF2ILA67Q, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742488371 (2h42m28.523421747s) [noise=true , address=104.36.82.9:4007 , piece_id=7D6VCXWVHACS3MKG2VT6RFPPDLKSMWQEZEDOS6FJ4A44YXULY5NA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742542194 (2h42m28.523475563s) [noise=true , address=104.36.82.13:4187 , piece_id=VOUFP55K63GOPO7352M43MOF6OOZAMGGNNFWOZBPNTIYMYIP47XA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 159.223.143.39:43136->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742614945 (2h42m28.523548309s) [noise=true , address=104.36.82.14:4223 , piece_id=S5EWQ2GVNMNPUM2QA72JM3Y55KVRLPX7XIGKMXJMTB2HUHHBAY4A, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:41060->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742668062 (2h42m28.523601432s) [noise=true , address=104.36.82.4:4804 , piece_id=KZBEMTFRGP7LJKTCF5ZVFBJH5EFFGQIMNTICJ7U3FEHOPRALWKXQ, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:46106->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742752541 (2h42m28.523685906s) [noise=true , address=104.36.82.10:4078 , piece_id=Q3UJWKRHCJEFIAHX7PTAHCKID3OSZIN5JAUOCOJGIRZ7AGQMHWEA, seg_pos={0 0}, upload=2489]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:59466->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:34:12.742794169 (2h42m28.523727553s) [seg_pos={0 0}, upload=2489]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:34:12.770005947 (2h42m28.550939315s) [upload=2489]: Done uploading segments... Jul 27 00:34:12.770043270 (2h42m28.550976637s) [upload=2489]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.029509443s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.X] duration: 30.029750509s FAIL (2025-07-27 00:34:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.029781575s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.................................................X..........] duration: 230.506961ms FAIL (2025-07-27 00:34:01 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 230.559805ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [..X..X......X.XX....X...X.................XX....X......XX.XX] duration: 30.0017946s FAIL (2025-07-27 00:33:57 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001816316s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.033159718s FAIL (2025-07-27 00:33:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== WR73Q5YM47RAPWB7U4VHUUGJM7GRB7ESRIJOICZQLUXQ, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.873653013 (2h40m48.654586394s) [noise=true , address=45.91.116.10:4093 , piece_id=63H2L4LKTHH5QYRV4TOP6ZQONA323DTCVWGPZBPLPIF6XV4TQ5YQ, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.874037487 (2h40m48.654970871s) [noise=true , address=79.127.201.214:4289 , piece_id=ZZK7PTB6CDICHZDG72FX57UJCXM5ICHBKQ5GOYE65HOEOBN5LKVA, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.874941234 (2h40m48.655874598s) [noise=true , address=104.36.82.51:4080 , piece_id=SPAJBTBULQT3RVPY6FUDQHK64I25GPWF3R5S5TTHF72DV5XVPXJA, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.875346425 (2h40m48.656279801s) [noise=true , address=104.36.82.52:4093 , piece_id=PWFRI4F2PJV4TZM4YCZD2VG6CU7E7NEAYFOBQVVPEF7MTVF7Y4TA, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.875723827 (2h40m48.656657212s) [noise=true , address=77.90.31.13:4631 , piece_id=2VRJ7AUGFDKW47E5JXAAUSU6OU7GBX2HPT43D3QQN232UHUFZKYQ, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.876233741 (2h40m48.657167196s) [noise=true , address=104.36.82.56:4245 , piece_id=YDIU2HDIEVCJIUB7NR33FJOQ4FRKQLDONMX4CTIRBOC2VK3HS5CQ, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.876656791 (2h40m48.657590177s) [noise=true , address=45.91.116.4:4964 , piece_id=3V6OQAYV3L5JPKDL6H3FRFDWXQOD4ZI5FE7PAOGSN3WFTVYBTQLA, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.877721857 (2h40m48.658655238s) [noise=true , address=79.127.201.215:4357 , piece_id=NDW4CGTHNJF7AUE4ORYMYFSYTIS65IUFBV6B7ZANCGDHH66U5VBA, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.878154730 (2h40m48.659088116s) [noise=true , address=64.49.13.140:4514 , piece_id=EAHKRZNAI4ZYTYG3DUDO563HPFOEOAOLTVC2TZJO23H3Q2YLWSTQ, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.878549266 (2h40m48.659482652s) [noise=true , address=79.127.205.234:4613 , piece_id=JNPUMNDJ3TOUURX4RJ4UUUH5X4Q4P5V46YRR3PPBY2UVKIIWGP5Q, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.878887423 (2h40m48.659820809s) [noise=true , address=79.127.201.214:4321 , piece_id=QGPKUL7XHWNN36HZVCCKTRUOZ6IOTHRUEK3TT7TP4R6Y6KFORUWA, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.881242445 (2h40m48.662175831s) [noise=true , address=156.146.43.225:4448 , piece_id=QSQWT6TYZI46MRXIXYN67BBVZBB257NGYWKDG4Y2SH4QXZSAMVHA, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.881751508 (2h40m48.662684892s) [noise=true , address=79.127.219.38:4007 , piece_id=5GNBYBDZSXHP4DGTGPEKXOFMAMYGWEQZJ246KTQPU6VNXVPS5JEQ, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.882091662 (2h40m48.663025055s) [noise=true , address=104.36.82.46:4488 , piece_id=MRJLUZQCZ6QBOEPZKXGXY3T7E7WG35HIZAWABFATEHD4XV2JIW2Q, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.882409721 (2h40m48.663343105s) [noise=true , address=79.127.205.245:4244 , piece_id=TZ2V2R34IL7D6LBYNVA4YCRIP4FTT6ASLJQ527OH7UIBFPBDFXWA, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:32.882707035 (2h40m48.663640415s) [noise=true , address=104.36.82.47:4517 , piece_id=3UTUJZV2EF5DPTJYUWHJIRJO52L5FTAJK6AVFSR5B5OBB7VHXKJQ, seg_pos={0 0}, upload=2473]: Uploading piece... Jul 27 00:32:33.003096411 (2h40m48.784029779s) [upload=2473]: Next returned nil segment Jul 27 00:32:33.003120023 (2h40m48.784053406s) [upload=2473]: Waiting for error group managing segments... Jul 27 00:32:33.045919038 (2h40m48.826852416s) [noise=true , address=79.127.219.38:4007 , piece_id=5GNBYBDZSXHP4DGTGPEKXOFMAMYGWEQZJ246KTQPU6VNXVPS5JEQ, seg_pos={0 0}, upload=2473]: Done uploading piece. err: <nil> Jul 27 00:32:33.061201994 (2h40m48.842135374s) [noise=true , address=79.127.201.214:4289 , piece_id=ZZK7PTB6CDICHZDG72FX57UJCXM5ICHBKQ5GOYE65HOEOBN5LKVA, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:1YfeEiw7zKAYgUcMqMLgEiS2t1gMqdwrpuAdTb2dZouuXRBSM4, address:79.127.201.214:4289): protocol: storage node overloaded, request limit: 2000 Jul 27 00:32:33.500067952 (2h40m49.281001317s) [noise=true , address=79.127.205.245:4244 , piece_id=TZ2V2R34IL7D6LBYNVA4YCRIP4FTT6ASLJQ527OH7UIBFPBDFXWA, seg_pos={0 0}, upload=2473]: Done uploading piece. err: <nil> Jul 27 00:32:33.509689087 (2h40m49.290622452s) [noise=true , address=79.127.205.246:4280 , piece_id=2K5GGHS6WR73Q5YM47RAPWB7U4VHUUGJM7GRB7ESRIJOICZQLUXQ, seg_pos={0 0}, upload=2473]: Done uploading piece. err: <nil> Jul 27 00:32:33.534479827 (2h40m49.31541319s) [noise=true , address=79.127.205.234:4613 , piece_id=JNPUMNDJ3TOUURX4RJ4UUUH5X4Q4P5V46YRR3PPBY2UVKIIWGP5Q, seg_pos={0 0}, upload=2473]: Done uploading piece. err: <nil> Jul 27 00:32:35.162221330 (2h40m50.943154699s) [noise=true , address=156.146.43.225:4448 , piece_id=QSQWT6TYZI46MRXIXYN67BBVZBB257NGYWKDG4Y2SH4QXZSAMVHA, seg_pos={0 0}, upload=2473]: Done uploading piece. err: <nil> Jul 27 00:32:35.330783593 (2h40m51.111716954s) [noise=true , address=79.127.201.214:4321 , piece_id=QGPKUL7XHWNN36HZVCCKTRUOZ6IOTHRUEK3TT7TP4R6Y6KFORUWA, seg_pos={0 0}, upload=2473]: Done uploading piece. err: <nil> Jul 27 00:32:35.392901462 (2h40m51.173834829s) [noise=true , address=64.49.13.140:4514 , piece_id=EAHKRZNAI4ZYTYG3DUDO563HPFOEOAOLTVC2TZJO23H3Q2YLWSTQ, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:1WsbDX2Zy8FYnEQtUsYFHSeDHj1DkkGyWBPaxzmReWFzaTejHn, address:64.49.13.140:4514): protocol: storage node overloaded, request limit: 2000 Jul 27 00:32:35.620578098 (2h40m51.401511463s) [noise=true , address=45.91.116.4:4964 , piece_id=3V6OQAYV3L5JPKDL6H3FRFDWXQOD4ZI5FE7PAOGSN3WFTVYBTQLA, seg_pos={0 0}, upload=2473]: Done uploading piece. err: <nil> Jul 27 00:32:45.927439038 (2h41m1.708372433s) [noise=true , address=45.91.116.10:4093 , piece_id=63H2L4LKTHH5QYRV4TOP6ZQONA323DTCVWGPZBPLPIF6XV4TQ5YQ, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:12bHetxKYyr24K2FqogjTPcsq7soJUQK7Tq6tHFZtvrNAmQnAxZ, address:45.91.116.10:4093): protocol: storage node overloaded, request limit: 2000 Jul 27 00:32:46.666851704 (2h41m2.447785069s) [noise=true , address=79.127.201.215:4357 , piece_id=NDW4CGTHNJF7AUE4ORYMYFSYTIS65IUFBV6B7ZANCGDHH66U5VBA, seg_pos={0 0}, upload=2473]: Done uploading piece. err: <nil> Jul 27 00:33:32.790751529 (2h41m48.571684909s) [noise=true , address=77.90.31.13:4631 , piece_id=2VRJ7AUGFDKW47E5JXAAUSU6OU7GBX2HPT43D3QQN232UHUFZKYQ, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:1aB5TGJszYj5bvk2cDbiQA7dcAoTxeYMsYgMWkkn1tACCxdtpz, address:77.90.31.13:4631): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:33:32.790845734 (2h41m48.571779114s) [noise=true , address=104.36.82.46:4488 , piece_id=MRJLUZQCZ6QBOEPZKXGXY3T7E7WG35HIZAWABFATEHD4XV2JIW2Q, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:1MU3fdv9cTkq4pUZ3h989STAG1wxrbhFgzmr6tcrfumZKHP9BZ, address:104.36.82.46:4488): protocol: write tcp 159.223.143.39:41762->104.36.82.46:4488: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:32.791293046 (2h41m48.572226424s) [noise=true , address=104.36.82.52:4093 , piece_id=PWFRI4F2PJV4TZM4YCZD2VG6CU7E7NEAYFOBQVVPEF7MTVF7Y4TA, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:12APDQdo9gz4mqC7TxxZE6zbf92LLP26iUYfpcpri4LyQn2ic4E, address:104.36.82.52:4093): protocol: write tcp 159.223.143.39:35142->104.36.82.52:4093: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:32.791347335 (2h41m48.572280715s) [noise=true , address=104.36.82.47:4517 , piece_id=3UTUJZV2EF5DPTJYUWHJIRJO52L5FTAJK6AVFSR5B5OBB7VHXKJQ, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:12HZjKTnjpeNU5B6muVHGrmWQS8GsyuYTW6cLxzNk5CGt3b8PJS, address:104.36.82.47:4517): protocol: write tcp 159.223.143.39:48942->104.36.82.47:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:32.791392426 (2h41m48.572325804s) [noise=true , address=104.36.82.51:4080 , piece_id=SPAJBTBULQT3RVPY6FUDQHK64I25GPWF3R5S5TTHF72DV5XVPXJA, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:1WXMdaUQnFnXfayz6CkmsK1wuLhWSpXjQQt2kZyVTVnv3qL44C, address:104.36.82.51:4080): protocol: write tcp 159.223.143.39:44202->104.36.82.51:4080: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:32.791536688 (2h41m48.572470067s) [noise=true , address=104.36.82.56:4245 , piece_id=YDIU2HDIEVCJIUB7NR33FJOQ4FRKQLDONMX4CTIRBOC2VK3HS5CQ, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:12YnxyH9TZVuVheQpH9NVRfFUeWHJEtV97wyEJafhCzGZSv13Fn, address:104.36.82.56:4245): protocol: write tcp 159.223.143.39:50252->104.36.82.56:4245: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:32.791749070 (2h41m48.572682448s) [noise=true , address=104.36.82.54:4192 , piece_id=3WY4POFOJ4A62SFUP3D5C2PEGT6Z7LFLEPKD4KCKSL25BN4LU2HA, seg_pos={0 0}, upload=2473]: Done uploading piece. err: ecclient: upload failed (node:12peSG85BzWEjq3FZ5xMqLE2gStDvy2dViRdYmDJz7Lq8yBfaKY, address:104.36.82.54:4192): protocol: write tcp 159.223.143.39:58184->104.36.82.54:4192: use of closed network connection; context deadline exceeded Jul 27 00:33:32.791943618 (2h41m48.572877012s) [seg_pos={0 0}, upload=2473]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 27 00:33:32.818918383 (2h41m48.599851767s) [upload=2473]: Done uploading segments... Jul 27 00:33:32.818965558 (2h41m48.599898942s) [upload=2473]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.033184486s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [.X........X...XX...............................X............] duration: 47.041645583s FAIL (2025-07-27 00:33:31 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 47.041699455s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.X] duration: 30.036525072s FAIL (2025-07-27 00:33:28 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== osed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:14.589446796 (2h40m30.370380174s) [seg_pos={0 0}, upload=2466]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:32:14.589537039 (2h40m30.370470418s) [upload=2466]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:32:14.589542609 (2h40m30.370475994s) [upload=2466]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:32:14.640249492 (2h40m30.421182861s) [upload=2466]: Done uploading segments... Jul 27 00:32:14.640270518 (2h40m30.421203886s) [upload=2466]: Upload finished. err: <nil> Jul 27 00:32:58.810974159 (2h41m14.591907543s) [upload=2481]: Starting upload Jul 27 00:32:58.811109781 (2h41m14.592043169s) [upload=2481]: Done starting upload Jul 27 00:32:58.811265564 (2h41m14.592198942s) [upload=2481]: Uploading segments... Jul 27 00:32:58.811695042 (2h41m14.592628422s) [upload=2481]: Got next segment. Inline: false Jul 27 00:32:58.811886987 (2h41m14.592820376s) [upload=2481]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:32:58.811897306 (2h41m14.592830689s) [upload=2481]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:32:58.860666254 (2h41m14.641599642s) [seg_pos={0 0}, upload=2481]: Begin upload segment... Jul 27 00:32:58.860753313 (2h41m14.641686696s) [seg_pos={0 0}, upload=2481]: Done begin upload segment. Jul 27 00:32:58.860816149 (2h41m14.641749548s) [noise=true , address=77.90.31.16:4734 , piece_id=Y6YU54NGQFFKDDYM3NLHHACBJU3O5POX6QFLOQK56IRSPQ6BO4SA, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.861286105 (2h41m14.642219492s) [noise=true , address=45.91.116.13:4153 , piece_id=F5DFROAFAGVTGEVWIIIYCM65AQIG5F5L6DJ27D2VJWBKGJP3HDKA, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.861698316 (2h41m14.6426317s) [noise=true , address=77.90.31.14:4663 , piece_id=IXAJPTO6ZLN5DPISKO6LOQ7FATM4LZV4MZETLMPMJS7MP62VO5TA, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.862003887 (2h41m14.642937267s) [noise=true , address=104.36.82.7:4912 , piece_id=MQQPZESGTLSRHZ57V4EYB3EZOILZHAF5HLZM67GXFIXIHUVSCJWQ, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.862312037 (2h41m14.643245454s) [noise=true , address=77.90.31.13:4628 , piece_id=MIOP42HZ77XLN4Z7FIVKQVRAXGHLB5QJUHSLNTLLY6LK4IYR3BVQ, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.862721254 (2h41m14.643654639s) [noise=true , address=77.90.31.15:4697 , piece_id=VOQWGYE32QQPU3RYAPN36N7ED4XM7JEIF6IFP7OOIB6YKV3IAVLA, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.863184293 (2h41m14.64411768s) [noise=true , address=77.90.31.19:4841 , piece_id=G4MUHCH3OG4YFYS722XM42ZUUIST6YGEWQGVQL26GXYCOJVCDLBA, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.863205444 (2h41m14.644138827s) [noise=true , address=104.36.82.5:4838 , piece_id=TII2TRXDFVNXUVYHNJGQBYVNJRKZFP5AYS26A74KN4HUUHLD63KQ, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.863713171 (2h41m14.644646559s) [noise=true , address=45.91.116.5:4297 , piece_id=EEO45A4W7UN7MNXUOT3PGZTQ5XYKWCI3Y4JNHFH2CAWEMGVHMUQQ, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.866569418 (2h41m14.647502805s) [noise=true , address=104.36.82.14:4221 , piece_id=7UH4FTXZYU7DGK2KDU6QKZT3JZQHJBB6BS6CDVFPYGU6KUHLKGWA, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.866984782 (2h41m14.647918159s) [noise=true , address=45.91.116.4:4939 , piece_id=RVEGNB5FPYKPMPXTZ7CZTKAFLKHWLSG6JQRI3D6EP2UTBAFPPSNQ, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.868444601 (2h41m14.649377988s) [noise=true , address=104.36.82.6:4873 , piece_id=KT3PFAPXSI2R7VKIOWG4BLQRLFYGHOYZWNVKJ2WFLHRUW74UO5ZA, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.868895493 (2h41m14.64982888s) [noise=true , address=45.91.116.7:4011 , piece_id=X7QDYAS43OY7H47XQ7APQ2BD6D2JE2IAFDIRDMNR377E5GTAWOSQ, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.869368113 (2h41m14.650301501s) [noise=true , address=45.91.116.8:4048 , piece_id=LVODTJLTSPJRFRA76P2WFKCPQFXTYSXZOUCRVK3NJLQNYYDUDFMA, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.869959041 (2h41m14.650892443s) [noise=true , address=45.91.116.11:4117 , piece_id=BTDJPKB62ETXNI5VGCJH7NAVSJPLHC5ATIU6VTEMQARXFHPODP7Q, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.870369000 (2h41m14.651302363s) [noise=true , address=45.91.116.8:4047 , piece_id=G5W7W2CY5ZPRFROVZWQQO76JE4276FG44FMH7YYYSISEHH2OPF7A, seg_pos={0 0}, upload=2481]: Uploading piece... Jul 27 00:32:58.973013076 (2h41m14.753946461s) [upload=2481]: Next returned nil segment Jul 27 00:32:58.973035721 (2h41m14.753969117s) [upload=2481]: Waiting for error group managing segments... Jul 27 00:32:58.977494008 (2h41m14.758427389s) [noise=true , address=45.91.116.4:4939 , piece_id=RVEGNB5FPYKPMPXTZ7CZTKAFLKHWLSG6JQRI3D6EP2UTBAFPPSNQ, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:32:58.978167121 (2h41m14.759100501s) [noise=true , address=45.91.116.11:4117 , piece_id=BTDJPKB62ETXNI5VGCJH7NAVSJPLHC5ATIU6VTEMQARXFHPODP7Q, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:32:58.978724285 (2h41m14.759657669s) [noise=true , address=45.91.116.7:4011 , piece_id=X7QDYAS43OY7H47XQ7APQ2BD6D2JE2IAFDIRDMNR377E5GTAWOSQ, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:32:58.979432936 (2h41m14.760366316s) [noise=true , address=45.91.116.8:4048 , piece_id=LVODTJLTSPJRFRA76P2WFKCPQFXTYSXZOUCRVK3NJLQNYYDUDFMA, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:32:58.981203510 (2h41m14.762136873s) [noise=true , address=77.90.31.14:4663 , piece_id=IXAJPTO6ZLN5DPISKO6LOQ7FATM4LZV4MZETLMPMJS7MP62VO5TA, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:32:58.983091363 (2h41m14.764024739s) [noise=true , address=45.91.116.8:4047 , piece_id=G5W7W2CY5ZPRFROVZWQQO76JE4276FG44FMH7YYYSISEHH2OPF7A, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:32:58.983281795 (2h41m14.764215174s) [noise=true , address=45.91.116.13:4153 , piece_id=F5DFROAFAGVTGEVWIIIYCM65AQIG5F5L6DJ27D2VJWBKGJP3HDKA, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:32:58.984095977 (2h41m14.765029343s) [noise=true , address=45.91.116.5:4297 , piece_id=EEO45A4W7UN7MNXUOT3PGZTQ5XYKWCI3Y4JNHFH2CAWEMGVHMUQQ, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:32:59.006215392 (2h41m14.787148773s) [noise=true , address=77.90.31.16:4734 , piece_id=Y6YU54NGQFFKDDYM3NLHHACBJU3O5POX6QFLOQK56IRSPQ6BO4SA, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:33:00.525382321 (2h41m16.306315703s) [noise=true , address=77.90.31.19:4841 , piece_id=G4MUHCH3OG4YFYS722XM42ZUUIST6YGEWQGVQL26GXYCOJVCDLBA, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:33:01.014116504 (2h41m16.795049889s) [noise=true , address=77.90.31.13:4628 , piece_id=MIOP42HZ77XLN4Z7FIVKQVRAXGHLB5QJUHSLNTLLY6LK4IYR3BVQ, seg_pos={0 0}, upload=2481]: Done uploading piece. err: <nil> Jul 27 00:33:28.779965871 (2h41m44.56089926s) [noise=true , address=104.36.82.5:4838 , piece_id=TII2TRXDFVNXUVYHNJGQBYVNJRKZFP5AYS26A74KN4HUUHLD63KQ, seg_pos={0 0}, upload=2481]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: write tcp 159.223.143.39:35344->104.36.82.5:4838: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:28.780050817 (2h41m44.560984198s) [noise=true , address=104.36.82.14:4221 , piece_id=7UH4FTXZYU7DGK2KDU6QKZT3JZQHJBB6BS6CDVFPYGU6KUHLKGWA, seg_pos={0 0}, upload=2481]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 159.223.143.39:43084->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:28.780117223 (2h41m44.561050604s) [noise=true , address=104.36.82.7:4912 , piece_id=MQQPZESGTLSRHZ57V4EYB3EZOILZHAF5HLZM67GXFIXIHUVSCJWQ, seg_pos={0 0}, upload=2481]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:42662->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:28.780175681 (2h41m44.56110907s) [noise=true , address=77.90.31.15:4697 , piece_id=VOQWGYE32QQPU3RYAPN36N7ED4XM7JEIF6IFP7OOIB6YKV3IAVLA, seg_pos={0 0}, upload=2481]: Done uploading piece. err: ecclient: upload failed (node:12AwQ11hGrJg7dmGK4WkceFmt6VSXLC7FXgXH159pCAe2muwps5, address:77.90.31.15:4697): protocol: write tcp 159.223.143.39:42492->77.90.31.15:4697: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:28.780234011 (2h41m44.56116739s) [noise=true , address=104.36.82.6:4873 , piece_id=KT3PFAPXSI2R7VKIOWG4BLQRLFYGHOYZWNVKJ2WFLHRUW74UO5ZA, seg_pos={0 0}, upload=2481]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:36250->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:33:28.780254899 (2h41m44.561188289s) [seg_pos={0 0}, upload=2481]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:33:28.813631340 (2h41m44.594564717s) [upload=2481]: Done uploading segments... Jul 27 00:33:28.813655080 (2h41m44.594588456s) [upload=2481]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.036557331s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 13 out of 60 is failed (threshold=3) [...X..X......X.XX....X...X.................XX....X......XX.X] duration: 30.000746848s FAIL (2025-07-27 00:33:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00077858s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [XXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X.] duration: 485.356412ms FAIL (2025-07-27 00:33:25 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 485.38246ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.X] duration: 30.037870024s FAIL (2025-07-27 00:33:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.037894755s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 250.181713ms FAIL (2025-07-27 00:33:04 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 250.233376ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [X.X........X...XX...............................X...........] duration: 40.126898939s FAIL (2025-07-27 00:32:44 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.126919107s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 207.555629ms FAIL (2025-07-27 00:32:35 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 207.589308ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.031851384s FAIL (2025-07-27 00:32:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== in Jul 27 00:31:32.755531495 (2h39m48.536464871s) [seg_pos={0 0}, upload=2455]: Begin upload segment... Jul 27 00:31:32.755580449 (2h39m48.536513818s) [seg_pos={0 0}, upload=2455]: Done begin upload segment. Jul 27 00:31:32.755604838 (2h39m48.536538217s) [noise=true , address=79.127.205.233:4642 , piece_id=LZ2LUU7D4PDW426J6YIINC6NMHFK5VPVZMFEW45EHKPFCRZKYI4A, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.756294030 (2h39m48.537227413s) [noise=true , address=64.49.13.138:4483 , piece_id=ANJOLBU75LIE3AQSATABBYCAPZCKMTVEYOD2BMLBW27CSLJS3PNA, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.756439527 (2h39m48.53737291s) [noise=true , address=77.90.31.13:4653 , piece_id=WDAJOMGIR2DNHXLXC72F5W4RLLYQO7QDHYOEWZNJK62AOMBE3SNQ, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.756688533 (2h39m48.537621918s) [noise=true , address=79.127.205.235:4657 , piece_id=XFEZ2C7PCLMTM2W4VKCS3IR2UKZPQJ7JT3RDRU7XTIH2IREUDX4Q, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.756929663 (2h39m48.537863043s) [noise=true , address=79.127.219.47:4048 , piece_id=NS263QCKRZQJ3EVHZXZJQGT2PTZFMUB7RVSIYF3QQAFLQOLSNEUQ, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.757055495 (2h39m48.537988876s) [noise=true , address=104.36.82.55:4210 , piece_id=NND3WRUQSG7XPAHAWI24NT6CDRYXYBDAH4TC24SIPJ6GER5DCFKA, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.757311437 (2h39m48.538244817s) [noise=true , address=104.36.82.12:4183 , piece_id=A5MP2V7YTA6WBFW4Q4MIW5A5YW4VRIIHY5N6GRZQLZMEED6C7P3Q, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.757350604 (2h39m48.53828398s) [noise=true , address=64.49.13.139:4478 , piece_id=BZC3LTB4TV333O3O3RKWP5A6Y7R7XO7V6YOSQEV72P47BD44ZQVQ, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.757644025 (2h39m48.5385774s) [noise=true , address=79.127.219.38:4009 , piece_id=7KNVGI5BUCDQAX7HN2BCH4JIGJRX2TBYZDCB4TBWKYA6EKGHWX6A, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.757699116 (2h39m48.538632498s) [noise=true , address=77.90.31.19:4844 , piece_id=VJW6FHTJSS73KP3MYTKS4BMY6R7SVQHJ76XACTHKUGHLRSDETD3A, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.758010437 (2h39m48.538943815s) [noise=true , address=104.36.82.11:4140 , piece_id=MYE7TBFT5DEIIV5TRLV6QNYDQCPUA7MADXBADHHLCQIC4X74UGVA, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.758079735 (2h39m48.539013115s) [noise=true , address=79.127.205.234:4620 , piece_id=3RK7LQ46VRABGW4GGH32BEI5NMBECTX7UOHFPRNUFULLJ35CI32A, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.758377486 (2h39m48.539310866s) [noise=true , address=77.90.31.11:4558 , piece_id=NHI5VU2HKDFVE2C5DPAXNLDPAZAXORXU5BYBFMYSJCPIT3UPWEYQ, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.758495831 (2h39m48.539429208s) [noise=true , address=104.36.82.46:4504 , piece_id=BHBUGIH6KTBP7TJOPYJYQ54HHPJVYZI2GBLJI3UZSRKY3VVQL5YA, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.758890307 (2h39m48.539823686s) [noise=true , address=77.90.31.15:4727 , piece_id=QEDGCWXWYKXL5HG74VAU7U7XDAICI3H5M7WD2X7BJ5AWZFM6LAMA, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.759349209 (2h39m48.540282597s) [noise=true , address=104.36.82.47:4550 , piece_id=D5VN3AY4VJQC7UQ2JX7NUUCMSLQ7FWQQLBPUAATJBTC4CWHVE2TA, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.759767619 (2h39m48.540700999s) [noise=true , address=79.127.201.215:4350 , piece_id=FNUF4VR3CMS5YNA57BM3CYRNQFDMNAMCA73AGB7QRKANQGGX3IQQ, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.760948126 (2h39m48.541881508s) [noise=true , address=156.146.43.225:4448 , piece_id=G4575CWREIA562T2YSTZGJXWMFH3VTXR3CYF4ZCNWGHFC7MYEE7A, seg_pos={0 0}, upload=2455]: Uploading piece... Jul 27 00:31:32.794929087 (2h39m48.575862464s) [noise=true , address=79.127.219.47:4048 , piece_id=NS263QCKRZQJ3EVHZXZJQGT2PTZFMUB7RVSIYF3QQAFLQOLSNEUQ, seg_pos={0 0}, upload=2455]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:31:32.832622020 (2h39m48.613555398s) [upload=2455]: Next returned nil segment Jul 27 00:31:32.832646129 (2h39m48.613579517s) [upload=2455]: Waiting for error group managing segments... Jul 27 00:31:32.884669000 (2h39m48.665602494s) [noise=true , address=77.90.31.11:4558 , piece_id=NHI5VU2HKDFVE2C5DPAXNLDPAZAXORXU5BYBFMYSJCPIT3UPWEYQ, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:31:32.945001603 (2h39m48.725934984s) [noise=true , address=79.127.219.38:4009 , piece_id=7KNVGI5BUCDQAX7HN2BCH4JIGJRX2TBYZDCB4TBWKYA6EKGHWX6A, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:31:33.350885835 (2h39m49.131819211s) [noise=true , address=79.127.205.233:4642 , piece_id=LZ2LUU7D4PDW426J6YIINC6NMHFK5VPVZMFEW45EHKPFCRZKYI4A, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:31:33.357512910 (2h39m49.138446282s) [noise=true , address=79.127.205.235:4657 , piece_id=XFEZ2C7PCLMTM2W4VKCS3IR2UKZPQJ7JT3RDRU7XTIH2IREUDX4Q, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:31:33.470874575 (2h39m49.251807954s) [noise=true , address=79.127.205.234:4620 , piece_id=3RK7LQ46VRABGW4GGH32BEI5NMBECTX7UOHFPRNUFULLJ35CI32A, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:31:35.421254041 (2h39m51.202187409s) [noise=true , address=156.146.43.225:4448 , piece_id=G4575CWREIA562T2YSTZGJXWMFH3VTXR3CYF4ZCNWGHFC7MYEE7A, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:31:38.980879644 (2h39m54.761813013s) [noise=true , address=64.49.13.139:4478 , piece_id=BZC3LTB4TV333O3O3RKWP5A6Y7R7XO7V6YOSQEV72P47BD44ZQVQ, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:31:39.757612496 (2h39m55.538545887s) [noise=true , address=64.49.13.138:4483 , piece_id=ANJOLBU75LIE3AQSATABBYCAPZCKMTVEYOD2BMLBW27CSLJS3PNA, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:31:46.744699323 (2h40m2.525632704s) [noise=true , address=77.90.31.19:4844 , piece_id=VJW6FHTJSS73KP3MYTKS4BMY6R7SVQHJ76XACTHKUGHLRSDETD3A, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:31:56.624825257 (2h40m12.405758621s) [noise=true , address=79.127.201.215:4350 , piece_id=FNUF4VR3CMS5YNA57BM3CYRNQFDMNAMCA73AGB7QRKANQGGX3IQQ, seg_pos={0 0}, upload=2455]: Done uploading piece. err: <nil> Jul 27 00:32:32.680428926 (2h40m48.46136231s) [noise=true , address=77.90.31.13:4653 , piece_id=WDAJOMGIR2DNHXLXC72F5W4RLLYQO7QDHYOEWZNJK62AOMBE3SNQ, seg_pos={0 0}, upload=2455]: Done uploading piece. err: ecclient: upload failed (node:12eaTnUhynF1XeuvMr68SjMyYKM9rLj5tShzpViHcMSjvMs9EBF, address:77.90.31.13:4653): protocol: expected piece hash; context deadline exceeded; EOF Jul 27 00:32:32.682632774 (2h40m48.463566162s) [noise=true , address=77.90.31.15:4727 , piece_id=QEDGCWXWYKXL5HG74VAU7U7XDAICI3H5M7WD2X7BJ5AWZFM6LAMA, seg_pos={0 0}, upload=2455]: Done uploading piece. err: ecclient: upload failed (node:1xoAbv6zJuG9LjyCdsCrTWQaCDyLrWyv8RCrMTiqNqazoEwD1M, address:77.90.31.15:4727): protocol: rpc: dial tcp 77.90.31.15:4727: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:32.682738939 (2h40m48.463672303s) [noise=true , address=104.36.82.46:4504 , piece_id=BHBUGIH6KTBP7TJOPYJYQ54HHPJVYZI2GBLJI3UZSRKY3VVQL5YA, seg_pos={0 0}, upload=2455]: Done uploading piece. err: ecclient: upload failed (node:1diUXMdXuNQNtEv5rYhk8sKkN3uQQCahjp9hQLTgRCknQ3B7D9, address:104.36.82.46:4504): protocol: rpc: dial tcp 104.36.82.46:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:32.682812664 (2h40m48.463746043s) [noise=true , address=104.36.82.12:4183 , piece_id=A5MP2V7YTA6WBFW4Q4MIW5A5YW4VRIIHY5N6GRZQLZMEED6C7P3Q, seg_pos={0 0}, upload=2455]: Done uploading piece. err: ecclient: upload failed (node:12QbZ2BghpxbZDkYtrcJFMf9FfcEmqXobs2xp9N8252UWHdDAGQ, address:104.36.82.12:4183): protocol: rpc: dial tcp 104.36.82.12:4183: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:32.682913196 (2h40m48.463846577s) [noise=true , address=104.36.82.11:4140 , piece_id=MYE7TBFT5DEIIV5TRLV6QNYDQCPUA7MADXBADHHLCQIC4X74UGVA, seg_pos={0 0}, upload=2455]: Done uploading piece. err: ecclient: upload failed (node:12AYQRkKcRprBgX1BdUwnd4MCQ1swYucsG9aYvLLEaskgrngkCB, address:104.36.82.11:4140): protocol: rpc: dial tcp 104.36.82.11:4140: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:32.682983903 (2h40m48.463917267s) [noise=true , address=104.36.82.47:4550 , piece_id=D5VN3AY4VJQC7UQ2JX7NUUCMSLQ7FWQQLBPUAATJBTC4CWHVE2TA, seg_pos={0 0}, upload=2455]: Done uploading piece. err: ecclient: upload failed (node:1Fz8JANjsHWJkHj7Y2JAkr7NTKqDtcKjRW4kDaYGLJYwgWnxbS, address:104.36.82.47:4550): protocol: rpc: dial tcp 104.36.82.47:4550: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:32.683091255 (2h40m48.464024617s) [noise=true , address=104.36.82.55:4210 , piece_id=NND3WRUQSG7XPAHAWI24NT6CDRYXYBDAH4TC24SIPJ6GER5DCFKA, seg_pos={0 0}, upload=2455]: Done uploading piece. err: ecclient: upload failed (node:12KuoSWvJCc3mfowPPSJkdCDewkrHZnG6gz1fWcmJRGR5wvHzLL, address:104.36.82.55:4210): protocol: rpc: dial tcp 104.36.82.55:4210: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:32.683115420 (2h40m48.46404881s) [seg_pos={0 0}, upload=2455]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:32:32.709912829 (2h40m48.490846206s) [upload=2455]: Done uploading segments... Jul 27 00:32:32.709942519 (2h40m48.490875896s) [upload=2455]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.031893925s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX.] duration: 2.572799671s FAIL (2025-07-27 00:32:24 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 2.572853134s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX.] duration: 2.511586473s FAIL (2025-07-27 00:32:14 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 2.129222200 (2h39m57.91015558s) [seg_pos={0 0}, upload=2448]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:31:42.156334822 (2h39m57.937268207s) [upload=2448]: Done uploading segments... Jul 27 00:31:42.156370339 (2h39m57.937303719s) [upload=2448]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:31:42.264407036 (2h39m58.045340421s) [upload=2460]: Starting upload Jul 27 00:31:42.264483387 (2h39m58.045416867s) [upload=2460]: Done starting upload Jul 27 00:31:42.264860765 (2h39m58.045794133s) [upload=2460]: Uploading segments... Jul 27 00:31:42.264920808 (2h39m58.045854196s) [upload=2460]: Got next segment. Inline: false Jul 27 00:31:42.264961330 (2h39m58.045894712s) [upload=2460]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:31:42.264966944 (2h39m58.045900329s) [upload=2460]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:31:42.293461093 (2h39m58.074394482s) [seg_pos={0 0}, upload=2460]: Begin upload segment... Jul 27 00:31:42.293548643 (2h39m58.074482013s) [seg_pos={0 0}, upload=2460]: Done begin upload segment. Jul 27 00:31:42.293578727 (2h39m58.074512117s) [noise=true , address=104.36.82.8:4970 , piece_id=FJB7JFLMXUPZPJO3BSTII4TVDSXTDABXK7IHEBEWUPY2FXMECBBQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.293927422 (2h39m58.074860811s) [noise=true , address=45.91.116.4:4939 , piece_id=EGG7N4M3YRMYYLLCSYKD6EIM2VY2FGIQCXEHQ2UNZNMEOQYIZK7A, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.294124034 (2h39m58.075057425s) [noise=true , address=45.91.116.12:4226 , piece_id=UE4KGLQZDEBRZGTW5WGZQOVOKRI75KYWGEXLVWLSN7Z5ASFDJGBA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.294737409 (2h39m58.075670798s) [noise=true , address=45.91.116.7:4009 , piece_id=3HPGGU324DPDQXYG2SDEHWA4Y6DWAE7BX6QJ6D5ZZTINEMFCWH5Q, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.294823688 (2h39m58.075757082s) [noise=true , address=77.90.31.18:4807 , piece_id=6UJ4IVWW4KOM2TWHNOO2KJRL4TTNLM2DO6NOCBRCKMRLPQYXJANQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.295336032 (2h39m58.07626942s) [noise=true , address=45.91.116.5:4298 , piece_id=6N6Q5ISLB3QUX3DVZ6EW2MWW2XMPAQNV3YB2WVQSFQ2HVWVNE3YA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.295509053 (2h39m58.076442443s) [noise=true , address=45.91.116.9:4263 , piece_id=S2SVSBKEGD55RM674IDLJEFUZNEPFK6PGRTQHLHM6XT4WHBE53TA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.295753727 (2h39m58.076687124s) [noise=true , address=104.36.82.2:4731 , piece_id=CXTVMN6DVKMP3SBREPOBRYEB6H653XWX4GL2QA7SRXSZKL7EGM5A, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.296251385 (2h39m58.077184773s) [noise=true , address=45.91.116.10:4083 , piece_id=VGFK6IU3365CATKBV7HNAC7VOH3LEHQKOBKHNMJCQZS44ZRNY45A, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.296332400 (2h39m58.077265787s) [noise=true , address=104.36.82.13:4186 , piece_id=SJS6IHJDMKBYQ4LPF6EDTLZYUI6C4FZVXCTCUDD2MAUKQ4E6JJ6Q, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.296643664 (2h39m58.077577054s) [noise=true , address=77.90.31.15:4699 , piece_id=WHASPWOMGQSJK77UCQKAPVWXIA2TNBH7SIU5RPAOAK4AEWNE6YSQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.296847000 (2h39m58.077780386s) [noise=true , address=45.91.116.7:4010 , piece_id=Y2EAWRYA2ENUMYMR2KCZLMYAW47LWDHQSJFOXQPAQRQVMABGLHZQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.297023987 (2h39m58.077957376s) [noise=true , address=104.36.82.6:4874 , piece_id=L26EDQYIMM7M25AWGNFSNEINT73ZAEPJPX5VHBBPTDPUUCIW7NGA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.297350006 (2h39m58.078283395s) [noise=true , address=45.91.116.7:4011 , piece_id=3TSX42H2AR4Y6GKFQ3GO6MYL5RV3L4Q6M4IWYVB6EJ3ZPR4FVOSA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.297947360 (2h39m58.078880751s) [noise=true , address=104.36.82.8:4972 , piece_id=G3454LTJPR5LFBE6ZRYWLNIVIHWKVCAGLONNOEATLQX4GNHDV3RQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.298453984 (2h39m58.079387378s) [noise=true , address=104.36.82.10:4079 , piece_id=ZT3WYT65FBQF2OLOARYQTUCUNMT2M2KPHTAUAREMRPTZ3J5Q5CWA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.408836511 (2h39m58.189769874s) [upload=2460]: Next returned nil segment Jul 27 00:31:42.408888035 (2h39m58.189821415s) [upload=2460]: Waiting for error group managing segments... Jul 27 00:31:42.417174109 (2h39m58.198107473s) [noise=true , address=45.91.116.7:4010 , piece_id=Y2EAWRYA2ENUMYMR2KCZLMYAW47LWDHQSJFOXQPAQRQVMABGLHZQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.417620922 (2h39m58.198554287s) [noise=true , address=45.91.116.9:4263 , piece_id=S2SVSBKEGD55RM674IDLJEFUZNEPFK6PGRTQHLHM6XT4WHBE53TA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.418285088 (2h39m58.199218454s) [noise=true , address=45.91.116.5:4298 , piece_id=6N6Q5ISLB3QUX3DVZ6EW2MWW2XMPAQNV3YB2WVQSFQ2HVWVNE3YA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.418846469 (2h39m58.199779834s) [noise=true , address=45.91.116.4:4939 , piece_id=EGG7N4M3YRMYYLLCSYKD6EIM2VY2FGIQCXEHQ2UNZNMEOQYIZK7A, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.419322117 (2h39m58.20025548s) [noise=true , address=45.91.116.12:4226 , piece_id=UE4KGLQZDEBRZGTW5WGZQOVOKRI75KYWGEXLVWLSN7Z5ASFDJGBA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.419928318 (2h39m58.200861684s) [noise=true , address=45.91.116.7:4011 , piece_id=3TSX42H2AR4Y6GKFQ3GO6MYL5RV3L4Q6M4IWYVB6EJ3ZPR4FVOSA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.420340054 (2h39m58.201273418s) [noise=true , address=45.91.116.7:4009 , piece_id=3HPGGU324DPDQXYG2SDEHWA4Y6DWAE7BX6QJ6D5ZZTINEMFCWH5Q, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.420923309 (2h39m58.201856673s) [noise=true , address=45.91.116.10:4083 , piece_id=VGFK6IU3365CATKBV7HNAC7VOH3LEHQKOBKHNMJCQZS44ZRNY45A, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:43.937493714 (2h39m59.718427082s) [noise=true , address=77.90.31.18:4807 , piece_id=6UJ4IVWW4KOM2TWHNOO2KJRL4TTNLM2DO6NOCBRCKMRLPQYXJANQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:32:12.221890828 (2h40m28.002824206s) [noise=true , address=104.36.82.10:4079 , piece_id=ZT3WYT65FBQF2OLOARYQTUCUNMT2M2KPHTAUAREMRPTZ3J5Q5CWA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.221959112 (2h40m28.002892501s) [noise=true , address=104.36.82.6:4874 , piece_id=L26EDQYIMM7M25AWGNFSNEINT73ZAEPJPX5VHBBPTDPUUCIW7NGA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222019962 (2h40m28.002953345s) [noise=true , address=77.90.31.15:4699 , piece_id=WHASPWOMGQSJK77UCQKAPVWXIA2TNBH7SIU5RPAOAK4AEWNE6YSQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 159.223.143.39:42112->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222065994 (2h40m28.002999372s) [noise=true , address=104.36.82.2:4731 , piece_id=CXTVMN6DVKMP3SBREPOBRYEB6H653XWX4GL2QA7SRXSZKL7EGM5A, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222138243 (2h40m28.003071624s) [noise=true , address=104.36.82.13:4186 , piece_id=SJS6IHJDMKBYQ4LPF6EDTLZYUI6C4FZVXCTCUDD2MAUKQ4E6JJ6Q, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222168712 (2h40m28.00310208s) [noise=true , address=104.36.82.8:4970 , piece_id=FJB7JFLMXUPZPJO3BSTII4TVDSXTDABXK7IHEBEWUPY2FXMECBBQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:32:12.222204212 (2h40m28.003137582s) [noise=true , address=104.36.82.8:4972 , piece_id=G3454LTJPR5LFBE6ZRYWLNIVIHWKVCAGLONNOEATLQX4GNHDV3RQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 159.223.143.39:51288->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222228959 (2h40m28.003162347s) [seg_pos={0 0}, upload=2460]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:32:12.247747435 (2h40m28.028680803s) [upload=2460]: Done uploading segments... Jul 27 00:32:12.247771546 (2h40m28.028704911s) [upload=2460]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 2.511610751s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..XX] duration: 30.027978284s FAIL (2025-07-27 00:32:12 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== 2.129222200 (2h39m57.91015558s) [seg_pos={0 0}, upload=2448]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:31:42.156334822 (2h39m57.937268207s) [upload=2448]: Done uploading segments... Jul 27 00:31:42.156370339 (2h39m57.937303719s) [upload=2448]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 27 00:31:42.264407036 (2h39m58.045340421s) [upload=2460]: Starting upload Jul 27 00:31:42.264483387 (2h39m58.045416867s) [upload=2460]: Done starting upload Jul 27 00:31:42.264860765 (2h39m58.045794133s) [upload=2460]: Uploading segments... Jul 27 00:31:42.264920808 (2h39m58.045854196s) [upload=2460]: Got next segment. Inline: false Jul 27 00:31:42.264961330 (2h39m58.045894712s) [upload=2460]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:31:42.264966944 (2h39m58.045900329s) [upload=2460]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:31:42.293461093 (2h39m58.074394482s) [seg_pos={0 0}, upload=2460]: Begin upload segment... Jul 27 00:31:42.293548643 (2h39m58.074482013s) [seg_pos={0 0}, upload=2460]: Done begin upload segment. Jul 27 00:31:42.293578727 (2h39m58.074512117s) [noise=true , address=104.36.82.8:4970 , piece_id=FJB7JFLMXUPZPJO3BSTII4TVDSXTDABXK7IHEBEWUPY2FXMECBBQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.293927422 (2h39m58.074860811s) [noise=true , address=45.91.116.4:4939 , piece_id=EGG7N4M3YRMYYLLCSYKD6EIM2VY2FGIQCXEHQ2UNZNMEOQYIZK7A, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.294124034 (2h39m58.075057425s) [noise=true , address=45.91.116.12:4226 , piece_id=UE4KGLQZDEBRZGTW5WGZQOVOKRI75KYWGEXLVWLSN7Z5ASFDJGBA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.294737409 (2h39m58.075670798s) [noise=true , address=45.91.116.7:4009 , piece_id=3HPGGU324DPDQXYG2SDEHWA4Y6DWAE7BX6QJ6D5ZZTINEMFCWH5Q, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.294823688 (2h39m58.075757082s) [noise=true , address=77.90.31.18:4807 , piece_id=6UJ4IVWW4KOM2TWHNOO2KJRL4TTNLM2DO6NOCBRCKMRLPQYXJANQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.295336032 (2h39m58.07626942s) [noise=true , address=45.91.116.5:4298 , piece_id=6N6Q5ISLB3QUX3DVZ6EW2MWW2XMPAQNV3YB2WVQSFQ2HVWVNE3YA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.295509053 (2h39m58.076442443s) [noise=true , address=45.91.116.9:4263 , piece_id=S2SVSBKEGD55RM674IDLJEFUZNEPFK6PGRTQHLHM6XT4WHBE53TA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.295753727 (2h39m58.076687124s) [noise=true , address=104.36.82.2:4731 , piece_id=CXTVMN6DVKMP3SBREPOBRYEB6H653XWX4GL2QA7SRXSZKL7EGM5A, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.296251385 (2h39m58.077184773s) [noise=true , address=45.91.116.10:4083 , piece_id=VGFK6IU3365CATKBV7HNAC7VOH3LEHQKOBKHNMJCQZS44ZRNY45A, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.296332400 (2h39m58.077265787s) [noise=true , address=104.36.82.13:4186 , piece_id=SJS6IHJDMKBYQ4LPF6EDTLZYUI6C4FZVXCTCUDD2MAUKQ4E6JJ6Q, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.296643664 (2h39m58.077577054s) [noise=true , address=77.90.31.15:4699 , piece_id=WHASPWOMGQSJK77UCQKAPVWXIA2TNBH7SIU5RPAOAK4AEWNE6YSQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.296847000 (2h39m58.077780386s) [noise=true , address=45.91.116.7:4010 , piece_id=Y2EAWRYA2ENUMYMR2KCZLMYAW47LWDHQSJFOXQPAQRQVMABGLHZQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.297023987 (2h39m58.077957376s) [noise=true , address=104.36.82.6:4874 , piece_id=L26EDQYIMM7M25AWGNFSNEINT73ZAEPJPX5VHBBPTDPUUCIW7NGA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.297350006 (2h39m58.078283395s) [noise=true , address=45.91.116.7:4011 , piece_id=3TSX42H2AR4Y6GKFQ3GO6MYL5RV3L4Q6M4IWYVB6EJ3ZPR4FVOSA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.297947360 (2h39m58.078880751s) [noise=true , address=104.36.82.8:4972 , piece_id=G3454LTJPR5LFBE6ZRYWLNIVIHWKVCAGLONNOEATLQX4GNHDV3RQ, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.298453984 (2h39m58.079387378s) [noise=true , address=104.36.82.10:4079 , piece_id=ZT3WYT65FBQF2OLOARYQTUCUNMT2M2KPHTAUAREMRPTZ3J5Q5CWA, seg_pos={0 0}, upload=2460]: Uploading piece... Jul 27 00:31:42.408836511 (2h39m58.189769874s) [upload=2460]: Next returned nil segment Jul 27 00:31:42.408888035 (2h39m58.189821415s) [upload=2460]: Waiting for error group managing segments... Jul 27 00:31:42.417174109 (2h39m58.198107473s) [noise=true , address=45.91.116.7:4010 , piece_id=Y2EAWRYA2ENUMYMR2KCZLMYAW47LWDHQSJFOXQPAQRQVMABGLHZQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.417620922 (2h39m58.198554287s) [noise=true , address=45.91.116.9:4263 , piece_id=S2SVSBKEGD55RM674IDLJEFUZNEPFK6PGRTQHLHM6XT4WHBE53TA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.418285088 (2h39m58.199218454s) [noise=true , address=45.91.116.5:4298 , piece_id=6N6Q5ISLB3QUX3DVZ6EW2MWW2XMPAQNV3YB2WVQSFQ2HVWVNE3YA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.418846469 (2h39m58.199779834s) [noise=true , address=45.91.116.4:4939 , piece_id=EGG7N4M3YRMYYLLCSYKD6EIM2VY2FGIQCXEHQ2UNZNMEOQYIZK7A, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.419322117 (2h39m58.20025548s) [noise=true , address=45.91.116.12:4226 , piece_id=UE4KGLQZDEBRZGTW5WGZQOVOKRI75KYWGEXLVWLSN7Z5ASFDJGBA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.419928318 (2h39m58.200861684s) [noise=true , address=45.91.116.7:4011 , piece_id=3TSX42H2AR4Y6GKFQ3GO6MYL5RV3L4Q6M4IWYVB6EJ3ZPR4FVOSA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.420340054 (2h39m58.201273418s) [noise=true , address=45.91.116.7:4009 , piece_id=3HPGGU324DPDQXYG2SDEHWA4Y6DWAE7BX6QJ6D5ZZTINEMFCWH5Q, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:42.420923309 (2h39m58.201856673s) [noise=true , address=45.91.116.10:4083 , piece_id=VGFK6IU3365CATKBV7HNAC7VOH3LEHQKOBKHNMJCQZS44ZRNY45A, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:31:43.937493714 (2h39m59.718427082s) [noise=true , address=77.90.31.18:4807 , piece_id=6UJ4IVWW4KOM2TWHNOO2KJRL4TTNLM2DO6NOCBRCKMRLPQYXJANQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: <nil> Jul 27 00:32:12.221890828 (2h40m28.002824206s) [noise=true , address=104.36.82.10:4079 , piece_id=ZT3WYT65FBQF2OLOARYQTUCUNMT2M2KPHTAUAREMRPTZ3J5Q5CWA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:12XgHpMHksVDkRiDokvccpMMCNXfvH29SBPspkNrLcZuPE2fdud, address:104.36.82.10:4079): protocol: rpc: dial tcp 104.36.82.10:4079: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.221959112 (2h40m28.002892501s) [noise=true , address=104.36.82.6:4874 , piece_id=L26EDQYIMM7M25AWGNFSNEINT73ZAEPJPX5VHBBPTDPUUCIW7NGA, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: rpc: dial tcp 104.36.82.6:4874: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222019962 (2h40m28.002953345s) [noise=true , address=77.90.31.15:4699 , piece_id=WHASPWOMGQSJK77UCQKAPVWXIA2TNBH7SIU5RPAOAK4AEWNE6YSQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: write tcp 159.223.143.39:42112->77.90.31.15:4699: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222065994 (2h40m28.002999372s) [noise=true , address=104.36.82.2:4731 , piece_id=CXTVMN6DVKMP3SBREPOBRYEB6H653XWX4GL2QA7SRXSZKL7EGM5A, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: rpc: dial tcp 104.36.82.2:4731: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222138243 (2h40m28.003071624s) [noise=true , address=104.36.82.13:4186 , piece_id=SJS6IHJDMKBYQ4LPF6EDTLZYUI6C4FZVXCTCUDD2MAUKQ4E6JJ6Q, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: rpc: dial tcp 104.36.82.13:4186: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222168712 (2h40m28.00310208s) [noise=true , address=104.36.82.8:4970 , piece_id=FJB7JFLMXUPZPJO3BSTII4TVDSXTDABXK7IHEBEWUPY2FXMECBBQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:1Fyc4xk3wSbE57sX7JZG3mex5Jwdqb7aey6PXSbCKHzkD7KUVe, address:104.36.82.8:4970): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:32:12.222204212 (2h40m28.003137582s) [noise=true , address=104.36.82.8:4972 , piece_id=G3454LTJPR5LFBE6ZRYWLNIVIHWKVCAGLONNOEATLQX4GNHDV3RQ, seg_pos={0 0}, upload=2460]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 159.223.143.39:51288->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:32:12.222228959 (2h40m28.003162347s) [seg_pos={0 0}, upload=2460]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 27 00:32:12.247747435 (2h40m28.028680803s) [upload=2460]: Done uploading segments... Jul 27 00:32:12.247771546 (2h40m28.028704911s) [upload=2460]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.027999544s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.X.X........X...XX...............................X..........] duration: 58.11963676s FAIL (2025-07-27 00:32:04 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 58.119663186s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [....X..X......X.XX....X...X.................XX....X......XX.] duration: 2.096086197s FAIL (2025-07-27 00:32:02 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.096142657s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 12 out of 60 is failed (threshold=3) [.....X..X......X.XX....X...X.................XX....X......XX] duration: 30.001743201s FAIL (2025-07-27 00:31:58 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001765835s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 38 out of 60 is failed (threshold=3) [X.XXXXX.XXXXXXXX...XXXXXX..XXX....X..X...X.X..X.XX.XXXXXX.XX] duration: 30.03478209s FAIL (2025-07-27 00:31:48 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.034802587s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXX.XX....XX..X.X...X.....XXXXXX...XX..X] duration: 30.032623612s FAIL (2025-07-27 00:31:42 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== 9 (2h38m27.729427932s) [noise=true , address=104.36.82.2:4732 , piece_id=FMVOOI66OYTQ5MMLQJDKV33SH5UBOBGH4XVSOIPXUZPHO5FIG7MQ, seg_pos={0 0}, upload=2433]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE): protocol: write tcp 159.223.143.39:54996->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:30:11.948517990 (2h38m27.729451357s) [seg_pos={0 0}, upload=2433]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:30:11.948618962 (2h38m27.729552332s) [upload=2433]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:30:11.948626030 (2h38m27.729559393s) [upload=2433]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:30:12.018855324 (2h38m27.799788716s) [upload=2433]: Done uploading segments... Jul 27 00:30:12.018897650 (2h38m27.799831049s) [upload=2433]: Upload finished. err: <nil> Jul 27 00:31:12.159695723 (2h39m27.940629087s) [upload=2448]: Starting upload Jul 27 00:31:12.159767937 (2h39m27.940701312s) [upload=2448]: Done starting upload Jul 27 00:31:12.160068267 (2h39m27.94100163s) [upload=2448]: Uploading segments... Jul 27 00:31:12.160432583 (2h39m27.941365963s) [upload=2448]: Got next segment. Inline: false Jul 27 00:31:12.160520325 (2h39m27.941453712s) [upload=2448]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:31:12.160546955 (2h39m27.941480339s) [upload=2448]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:31:12.197153293 (2h39m27.978086678s) [seg_pos={0 0}, upload=2448]: Begin upload segment... Jul 27 00:31:12.197241485 (2h39m27.978174874s) [noise=true , address=45.91.116.10:4082 , piece_id=2IN4MTZPW6BQ6ZMBJYM45SEBNFMHKDAXKIAZQLJP6PSBQUJ24U2Q, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.202968821 (2h39m27.98390221s) [seg_pos={0 0}, upload=2448]: Done begin upload segment. Jul 27 00:31:12.203093298 (2h39m27.984026687s) [noise=true , address=77.90.31.14:4662 , piece_id=OU2W5SJZ7NU5ZTJQQHOIPN7LCEAQT5OGBW2CKJ4USKGWKDAYMFKA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.203579409 (2h39m27.984512799s) [noise=true , address=77.90.31.14:4663 , piece_id=FRZPEUEMNTJDX27WADLUBC22CXE74JPSFOUDNNL7NJ4TMCNZMF5A, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.204015604 (2h39m27.984948989s) [noise=true , address=45.91.116.12:4225 , piece_id=LFZYMP5GSIOZDCYFHPMY64PLVZC5CBCNK7DO2M77GCL4JKA46TNA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.204401073 (2h39m27.985334462s) [noise=true , address=77.90.31.13:4627 , piece_id=AZSM4QUF77TECDU77PFUKT6YALD6Y4PUJBTJSAHY7UYFX46WLQMA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.205013220 (2h39m27.985946607s) [noise=true , address=45.91.116.13:4155 , piece_id=3MJY3OVR34AU53LQFYLYCJ7IQVOGM3SKN6VRSCIZF6XHVDKJQI3Q, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.205444844 (2h39m27.986378234s) [noise=true , address=104.36.82.5:4839 , piece_id=FABXNVRH7NJS5V6OJ3EBQVXBIUTVYPBRQC6OXECMZANGYNWMNMZQ, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.205810422 (2h39m27.986743808s) [noise=true , address=45.91.116.5:4298 , piece_id=M6EQY4PDGWSZOC7SXMEJXT3FGKOH5KLJ4U4L5DECVHXX6HSKNAWA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.206184793 (2h39m27.987118178s) [noise=true , address=77.90.31.15:4698 , piece_id=E24T5XO47376KU65ACGW2VJIOQ57YXKZUN6YVQDPVCQNAG5RZH2A, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.206619846 (2h39m27.987553232s) [noise=true , address=45.91.116.5:4299 , piece_id=ODBKEKETYG7L3UKMXYTFJAINZIOHG4YGDUSXM6JKMBDYKI4VFEXA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.207165081 (2h39m27.988098471s) [noise=true , address=45.91.116.14:4191 , piece_id=PO2GPLLZU7CQ3AH7KK57TRRZPAPX26JB7B3O4XA3ZO3MNOMK4RKQ, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.207850036 (2h39m27.988783423s) [noise=true , address=104.36.82.13:4185 , piece_id=WRNHR7X33VHZMMDCLHBIJFXU3HYNJT5FLATA34JMOZFNIC5ATDLQ, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.209818931 (2h39m27.990752331s) [noise=true , address=45.91.116.6:4973 , piece_id=PJ6SFQ37O525LBTWMFVXPUTZODL43XO5O5X7PCCZ4ONKBZNTQL7Q, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.210267025 (2h39m27.991200412s) [noise=true , address=45.91.116.8:4045 , piece_id=4GLSLSJ4SEYP6Z4LUZKC3UUT44KEZXUJRTMGVR23DKLV5M3PIK5A, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.210661756 (2h39m27.991595142s) [noise=true , address=104.36.82.8:4969 , piece_id=CA667UCM3VDDIPWLRZ6URFSSJOJXMUAQX4KJWJSYLL3DJBXWK7CA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.214500450 (2h39m27.995433836s) [noise=true , address=77.90.31.11:4553 , piece_id=NFR6VI445RLIQ7AO6GCEMASE5MKGQYHMMRWJCCLNORO3C6YIXRXA, seg_pos={0 0}, upload=2448]: Uploading piece... Jul 27 00:31:12.322882780 (2h39m28.103816164s) [upload=2448]: Next returned nil segment Jul 27 00:31:12.322902688 (2h39m28.103836074s) [upload=2448]: Waiting for error group managing segments... Jul 27 00:31:12.327209057 (2h39m28.108142436s) [noise=true , address=45.91.116.5:4299 , piece_id=ODBKEKETYG7L3UKMXYTFJAINZIOHG4YGDUSXM6JKMBDYKI4VFEXA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.328683217 (2h39m28.109616595s) [noise=true , address=45.91.116.6:4973 , piece_id=PJ6SFQ37O525LBTWMFVXPUTZODL43XO5O5X7PCCZ4ONKBZNTQL7Q, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.329209607 (2h39m28.110142971s) [noise=true , address=45.91.116.12:4225 , piece_id=LFZYMP5GSIOZDCYFHPMY64PLVZC5CBCNK7DO2M77GCL4JKA46TNA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.330333791 (2h39m28.111267177s) [noise=true , address=77.90.31.14:4662 , piece_id=OU2W5SJZ7NU5ZTJQQHOIPN7LCEAQT5OGBW2CKJ4USKGWKDAYMFKA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.331153586 (2h39m28.112086957s) [noise=true , address=45.91.116.10:4082 , piece_id=2IN4MTZPW6BQ6ZMBJYM45SEBNFMHKDAXKIAZQLJP6PSBQUJ24U2Q, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.332133429 (2h39m28.113066797s) [noise=true , address=77.90.31.14:4663 , piece_id=FRZPEUEMNTJDX27WADLUBC22CXE74JPSFOUDNNL7NJ4TMCNZMF5A, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.332833107 (2h39m28.113766485s) [noise=true , address=45.91.116.8:4045 , piece_id=4GLSLSJ4SEYP6Z4LUZKC3UUT44KEZXUJRTMGVR23DKLV5M3PIK5A, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.333485917 (2h39m28.11441928s) [noise=true , address=45.91.116.5:4298 , piece_id=M6EQY4PDGWSZOC7SXMEJXT3FGKOH5KLJ4U4L5DECVHXX6HSKNAWA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.334923028 (2h39m28.115856408s) [noise=true , address=45.91.116.13:4155 , piece_id=3MJY3OVR34AU53LQFYLYCJ7IQVOGM3SKN6VRSCIZF6XHVDKJQI3Q, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.377488636 (2h39m28.158422001s) [noise=true , address=45.91.116.14:4191 , piece_id=PO2GPLLZU7CQ3AH7KK57TRRZPAPX26JB7B3O4XA3ZO3MNOMK4RKQ, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:12.442882082 (2h39m28.22381546s) [noise=true , address=77.90.31.11:4553 , piece_id=NFR6VI445RLIQ7AO6GCEMASE5MKGQYHMMRWJCCLNORO3C6YIXRXA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:13.842044043 (2h39m29.62297741s) [noise=true , address=77.90.31.13:4627 , piece_id=AZSM4QUF77TECDU77PFUKT6YALD6Y4PUJBTJSAHY7UYFX46WLQMA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: <nil> Jul 27 00:31:42.128926436 (2h39m57.909859814s) [noise=true , address=104.36.82.5:4839 , piece_id=FABXNVRH7NJS5V6OJ3EBQVXBIUTVYPBRQC6OXECMZANGYNWMNMZQ, seg_pos={0 0}, upload=2448]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: rpc: dial tcp 104.36.82.5:4839: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:42.129053492 (2h39m57.909986871s) [noise=true , address=104.36.82.13:4185 , piece_id=WRNHR7X33VHZMMDCLHBIJFXU3HYNJT5FLATA34JMOZFNIC5ATDLQ, seg_pos={0 0}, upload=2448]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:42.129130004 (2h39m57.910063384s) [noise=true , address=104.36.82.8:4969 , piece_id=CA667UCM3VDDIPWLRZ6URFSSJOJXMUAQX4KJWJSYLL3DJBXWK7CA, seg_pos={0 0}, upload=2448]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: rpc: dial tcp 104.36.82.8:4969: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:42.129201387 (2h39m57.910134767s) [noise=true , address=77.90.31.15:4698 , piece_id=E24T5XO47376KU65ACGW2VJIOQ57YXKZUN6YVQDPVCQNAG5RZH2A, seg_pos={0 0}, upload=2448]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: write tcp 159.223.143.39:40516->77.90.31.15:4698: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:31:42.129222200 (2h39m57.91015558s) [seg_pos={0 0}, upload=2448]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 27 00:31:42.156334822 (2h39m57.937268207s) [upload=2448]: Done uploading segments... Jul 27 00:31:42.156370339 (2h39m57.937303719s) [upload=2448]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.032649518s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 223.567255ms FAIL (2025-07-27 00:31:36 UTC) orbiter warning: unable to find key in bucket: sj://uplink/orbiter_us_0/1K Duration: 223.60094ms core uplink us1 vivint

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)