Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..XX] duration: 30.029461798s FAIL (2025-07-27 00:08:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.029491526s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 60 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029778088s FAIL (2025-07-27 00:07:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== ad=2068]: Starting upload Jul 27 00:06:58.373532907 (2h15m14.154466276s) [upload=2068]: Done starting upload Jul 27 00:06:58.373651073 (2h15m14.154584441s) [upload=2068]: Uploading segments... Jul 27 00:06:58.373937665 (2h15m14.154871036s) [upload=2068]: Got next segment. Inline: false Jul 27 00:06:58.373984986 (2h15m14.154918354s) [upload=2068]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:06:58.373991665 (2h15m14.154925032s) [upload=2068]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:06:58.413242534 (2h15m14.19417591s) [seg_pos={0 0}, upload=2068]: Begin upload segment... Jul 27 00:06:58.413320530 (2h15m14.194253909s) [noise=true , address=79.127.219.38:4010 , piece_id=PVX626CPFPWPP5Z34RLZF6HT4LBFEHKDIFINFGUW7QLVNIUK462Q, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.413934784 (2h15m14.194868173s) [noise=true , address=79.127.205.245:4240 , piece_id=NW7QFPDHNP3XDREHQZDGVGOWBL4NSRPF432QNKNZHJ7FRBLLEDOA, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.414450868 (2h15m14.195384252s) [noise=true , address=64.49.13.138:4488 , piece_id=WPNNCSMDIJQJK47Z7TGWXFM35NZL4STMCF33EV6PCCCPBE7W25QQ, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.414798904 (2h15m14.195732291s) [seg_pos={0 0}, upload=2068]: Done begin upload segment. Jul 27 00:06:58.414830152 (2h15m14.195763547s) [noise=true , address=104.36.82.47:4522 , piece_id=EBH2HJFBR5KRPGLXWP4RXMHVYURRVLITFXHDSY6WBBJHDCJZNFOQ, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.415225560 (2h15m14.196158942s) [noise=true , address=77.90.31.11:4573 , piece_id=ZQROWPUM5TX76GFAT2NMHT4SVJ4N25AMOBPO4ZBNZ4EEI2Z3RM7A, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.415951604 (2h15m14.19688499s) [noise=true , address=79.127.219.51:4210 , piece_id=QAL4UEWTVB3TG2G2PP33C3REG4X32PYNG5GMZX5FZBYWRIVM5TEA, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.416156711 (2h15m14.197090103s) [noise=true , address=79.127.219.48:4074 , piece_id=77WFCSA62AP3YRQZYUAU4NEQ2VO32OLDUCK22OZYHKWMG65AAPNQ, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.416834350 (2h15m14.197767736s) [noise=true , address=79.127.205.234:4610 , piece_id=5UKXEIQUIJ36ESPZGUEYJYBOHX6KJ4J6TWZBYJ6NTDTG2SGLQKRA, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.417407625 (2h15m14.198340993s) [noise=true , address=104.36.82.53:4138 , piece_id=BYFFJJH7OWDW3RL2BLNF4JJJDDIJNKZ7N3QAS5DKBUKYEVFRFTXA, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.418647613 (2h15m14.199581s) [noise=true , address=64.49.13.137:4502 , piece_id=XHI442SNUIFHZEZB367WTQRCDPFMJ6YOG77QNOK5BADS4FMFQH4A, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.419720953 (2h15m14.20065434s) [noise=true , address=79.127.219.47:4048 , piece_id=I3ZMDYNVLI2NXDM73TZOHCL3NCJEJEVQYFGY7RFZURSC4O4QVT3A, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.420409375 (2h15m14.201342756s) [noise=true , address=104.36.82.54:4175 , piece_id=CYTWMWDQKJOWD2DWMYZ64F66RIIN7ANKVPL5EZUFTXRQF4UTDBLA, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.421117273 (2h15m14.202050661s) [noise=true , address=45.91.116.9:4277 , piece_id=B6UDYRSHLLNGA3M22QURXGQ7TYHZY7ER6VI5XPO33QADHP5ZZP5Q, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.421667155 (2h15m14.202600539s) [noise=true , address=79.127.205.233:4655 , piece_id=2CHZN3DRC4MH3ZWW4DDC5EWAQQ7XSPL5ALWJFEEDLGFRNANDNIOA, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.421730342 (2h15m14.202663705s) [noise=true , address=104.36.82.21:4497 , piece_id=5C2ZJDDTAVAORHXHE4J4SGGOQL7F55RY426Z7SMJTM5PNXWRQ67A, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.423126966 (2h15m14.204060354s) [noise=true , address=104.36.82.46:4496 , piece_id=QYXV7PQLHVMDT5GH3OTPHBYW3NCK3WNJSQT7ZVGXNBUUXABFSINA, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.423236450 (2h15m14.204169813s) [noise=true , address=77.90.31.16:4736 , piece_id=YQF5BNYVS4ECPTNZ6VQ47KXR24X5ZDO2KBEU6LCJYRQPADRSJKMQ, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.424063221 (2h15m14.204996584s) [noise=true , address=79.127.201.215:4360 , piece_id=ZFD67RCH337WY4YZEFNOLELSQUXHFVPWE33GXQ4DIG6AXRITG46A, seg_pos={0 0}, upload=2068]: Uploading piece... Jul 27 00:06:58.508849014 (2h15m14.289782383s) [upload=2068]: Next returned nil segment Jul 27 00:06:58.508872409 (2h15m14.289805808s) [upload=2068]: Waiting for error group managing segments... Jul 27 00:06:58.513728587 (2h15m14.294661952s) [noise=true , address=79.127.219.38:4010 , piece_id=PVX626CPFPWPP5Z34RLZF6HT4LBFEHKDIFINFGUW7QLVNIUK462Q, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:06:58.533893760 (2h15m14.31482713s) [noise=true , address=77.90.31.16:4736 , piece_id=YQF5BNYVS4ECPTNZ6VQ47KXR24X5ZDO2KBEU6LCJYRQPADRSJKMQ, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:06:58.553290858 (2h15m14.334224247s) [noise=true , address=77.90.31.11:4573 , piece_id=ZQROWPUM5TX76GFAT2NMHT4SVJ4N25AMOBPO4ZBNZ4EEI2Z3RM7A, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:06:58.610490363 (2h15m14.391423755s) [noise=true , address=79.127.219.47:4048 , piece_id=I3ZMDYNVLI2NXDM73TZOHCL3NCJEJEVQYFGY7RFZURSC4O4QVT3A, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:06:59.029814337 (2h15m14.810747703s) [noise=true , address=79.127.205.245:4240 , piece_id=NW7QFPDHNP3XDREHQZDGVGOWBL4NSRPF432QNKNZHJ7FRBLLEDOA, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:06:59.058372704 (2h15m14.839306069s) [noise=true , address=79.127.205.233:4655 , piece_id=2CHZN3DRC4MH3ZWW4DDC5EWAQQ7XSPL5ALWJFEEDLGFRNANDNIOA, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:06:59.099820554 (2h15m14.880753937s) [noise=true , address=64.49.13.137:4502 , piece_id=XHI442SNUIFHZEZB367WTQRCDPFMJ6YOG77QNOK5BADS4FMFQH4A, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:06:59.425431871 (2h15m15.20636525s) [noise=true , address=79.127.205.234:4610 , piece_id=5UKXEIQUIJ36ESPZGUEYJYBOHX6KJ4J6TWZBYJ6NTDTG2SGLQKRA, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:07:00.319004901 (2h15m16.099938282s) [noise=true , address=79.127.219.51:4210 , piece_id=QAL4UEWTVB3TG2G2PP33C3REG4X32PYNG5GMZX5FZBYWRIVM5TEA, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:07:01.785238153 (2h15m17.566171515s) [noise=true , address=79.127.219.48:4074 , piece_id=77WFCSA62AP3YRQZYUAU4NEQ2VO32OLDUCK22OZYHKWMG65AAPNQ, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:07:08.239746995 (2h15m24.020680361s) [noise=true , address=64.49.13.138:4488 , piece_id=WPNNCSMDIJQJK47Z7TGWXFM35NZL4STMCF33EV6PCCCPBE7W25QQ, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:07:17.856709238 (2h15m33.637642602s) [noise=true , address=45.91.116.9:4277 , piece_id=B6UDYRSHLLNGA3M22QURXGQ7TYHZY7ER6VI5XPO33QADHP5ZZP5Q, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:07:20.148635163 (2h15m35.929568551s) [noise=true , address=79.127.201.215:4360 , piece_id=ZFD67RCH337WY4YZEFNOLELSQUXHFVPWE33GXQ4DIG6AXRITG46A, seg_pos={0 0}, upload=2068]: Done uploading piece. err: <nil> Jul 27 00:07:58.337825465 (2h16m14.11875885s) [noise=true , address=104.36.82.53:4138 , piece_id=BYFFJJH7OWDW3RL2BLNF4JJJDDIJNKZ7N3QAS5DKBUKYEVFRFTXA, seg_pos={0 0}, upload=2068]: Done uploading piece. err: ecclient: upload failed (node:1YUGGJg6mWeQMmzSNWtLFWAaWwaKWfEf8wxXs1jwiZaA9nhsr3, address:104.36.82.53:4138): protocol: rpc: dial tcp 104.36.82.53:4138: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:58.337936421 (2h16m14.118869798s) [noise=true , address=104.36.82.21:4497 , piece_id=5C2ZJDDTAVAORHXHE4J4SGGOQL7F55RY426Z7SMJTM5PNXWRQ67A, seg_pos={0 0}, upload=2068]: Done uploading piece. err: ecclient: upload failed (node:12ZBe4we7VeFdthqRxyy1V9uQrA4r8ouNX68xqsk5bD26dkf2Gd, address:104.36.82.21:4497): protocol: rpc: dial tcp 104.36.82.21:4497: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:58.338008663 (2h16m14.118942045s) [noise=true , address=104.36.82.47:4522 , piece_id=EBH2HJFBR5KRPGLXWP4RXMHVYURRVLITFXHDSY6WBBJHDCJZNFOQ, seg_pos={0 0}, upload=2068]: Done uploading piece. err: ecclient: upload failed (node:1zGZe6e37uop4y622fU1GbspMPVEZURS3FbMrf7je6VT9TYGFr, address:104.36.82.47:4522): protocol: rpc: dial tcp 104.36.82.47:4522: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:58.338077348 (2h16m14.119010725s) [noise=true , address=104.36.82.54:4175 , piece_id=CYTWMWDQKJOWD2DWMYZ64F66RIIN7ANKVPL5EZUFTXRQF4UTDBLA, seg_pos={0 0}, upload=2068]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:58.338155524 (2h16m14.119088894s) [noise=true , address=104.36.82.46:4496 , piece_id=QYXV7PQLHVMDT5GH3OTPHBYW3NCK3WNJSQT7ZVGXNBUUXABFSINA, seg_pos={0 0}, upload=2068]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: write tcp 159.223.143.39:59222->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:07:58.338186479 (2h16m14.119119858s) [seg_pos={0 0}, upload=2068]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 27 00:07:58.365161557 (2h16m14.146094941s) [upload=2068]: Done uploading segments... Jul 27 00:07:58.365198927 (2h16m14.146132307s) [upload=2068]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.029802245s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....XXX...XX.....X.XXX...............X..X......X.XX....X...X] duration: 30.004359571s FAIL (2025-07-27 00:07:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.004377516s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..X] duration: 30.026333626s FAIL (2025-07-27 00:07:33 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.026390667s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.....X...X....XX..............X.X........X...XX.............] duration: 24.005517359s FAIL (2025-07-27 00:07:25 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 24.005541934s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [......X...X....XX..............X.X........X...XX............] duration: 52.433451111s FAIL (2025-07-27 00:07:00 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.433482642s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 59 out of 60 is failed (threshold=3) [.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.029601401s FAIL (2025-07-27 00:06:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 05:58.285655896 (2h14m14.066589275s) [upload=2055]: Got next segment. Inline: false Jul 27 00:05:58.285750294 (2h14m14.066683671s) [upload=2055]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:05:58.285759073 (2h14m14.066692453s) [upload=2055]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:05:58.325522388 (2h14m14.106455765s) [seg_pos={0 0}, upload=2055]: Begin upload segment... Jul 27 00:05:58.325582098 (2h14m14.106515463s) [seg_pos={0 0}, upload=2055]: Done begin upload segment. Jul 27 00:05:58.325614332 (2h14m14.106547729s) [noise=true , address=64.49.13.138:4488 , piece_id=CZBKAYLTRLOTMU2QA2Q5THD4YVFGQ6SGPFQCOCCWE6MZ55S42GZQ, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.325647583 (2h14m14.106580973s) [noise=true , address=104.36.82.46:4513 , piece_id=RFUY2S6J5LTKLMCPJ5XUALZKVRLTVGQUXB2N7DUUL6LNXT2LRRAQ, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.326034066 (2h14m14.106967445s) [noise=true , address=79.127.205.245:4227 , piece_id=NMI7IN2I546RHHWPK3G6YGFII6EMDMLKC7KH5QQ6AXQ5PVXC4DIQ, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.326248547 (2h14m14.107181933s) [noise=true , address=104.36.82.42:4807 , piece_id=QHVUNNT6BC5PUJTUF2GK3NZVUS3QHMEGHR5Q3HSKY33GCPU3XXFQ, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.326353861 (2h14m14.107287241s) [noise=true , address=79.127.205.233:4624 , piece_id=MM2AMS7F65ACLP7K7T4GPSLWSCZRAFYPSEAEICMVBIH27WPQUAYA, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.326652705 (2h14m14.107586085s) [noise=true , address=104.36.82.47:4518 , piece_id=GYTLS7JW2CE67UI6XMAZBHSFDGO4X2N7PHBVRFHUNJ6AWXRAG33A, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.326757970 (2h14m14.107691357s) [noise=true , address=104.36.82.56:4263 , piece_id=FX7AFYY7VA6LXBQXQBB6K6QFVDM7SSOL2M4HJW7ZZS4JNKODWEYQ, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.326946463 (2h14m14.107879839s) [noise=true , address=104.36.82.41:4792 , piece_id=3VOW6ULEXAMKY5XKUPN2M2OF5QOC7UKWLNO5MGCQRQCLXOKL7DIQ, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.327291335 (2h14m14.108224719s) [noise=true , address=79.127.205.235:4678 , piece_id=ROLUJUTRCRCPUCH2UDR6CNL75TGR6OK4VP5DLJ5YZBG3SMBVYOIA, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.327302756 (2h14m14.108236132s) [noise=true , address=77.90.31.12:4599 , piece_id=VUO6TD6RHSC2AS4W35SJE76FVPHDUAJSFIKYU3O4PKOD3A2BEEQA, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.327622421 (2h14m14.108555804s) [noise=true , address=79.127.220.101:4041 , piece_id=SNPFNKV2APLHHCZOO6NZFJ5GYLSDHA6PXZ7Q4OY7VSOFLJMLP5MA, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.327835159 (2h14m14.108768535s) [noise=true , address=79.127.220.101:4048 , piece_id=5JCBUFMQ26LZO3V63U7TDF6EIC3CVMNLH2F4CXAWIJEP2X3CBTNA, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.327999948 (2h14m14.108933328s) [noise=true , address=77.90.31.13:4629 , piece_id=YCL43MLCDT7H5JEXZTJPFZPBYPCQDU5OBCZUDF6BWVCL6WH7D3WA, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.328534828 (2h14m14.10946821s) [noise=true , address=79.127.205.246:4284 , piece_id=MOZVL2IO5BVCGGXMCEHBR44N75Q6X4UFGKYGX7B5PJHBIRSV3PIA, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.329101753 (2h14m14.110035139s) [noise=true , address=77.90.31.11:4581 , piece_id=L5FL6RZAKVEEVE3FKYVSKMMWSZUE4AGCSIDPIPBUVNXOJ4N2PTMA, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.330680714 (2h14m14.111614101s) [noise=true , address=79.127.201.215:4351 , piece_id=HWJEB3QRKBAYY77XPW43B5QLIGZO7IKROYY7BRLBJZ2WJI2UR62A, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.330787939 (2h14m14.111721323s) [noise=true , address=79.127.205.234:4607 , piece_id=IBB23QLHILMWRMWRCF4525OXXHOAD5F2WBGACMEQFSAU2RQKKMLQ, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.332921640 (2h14m14.113855013s) [noise=true , address=77.90.31.21:4896 , piece_id=C4KLG3F3ZFKGDLCVQ5TYMLLDY435RYGFKH3MRBMADOHP7ZK33LXA, seg_pos={0 0}, upload=2055]: Uploading piece... Jul 27 00:05:58.376280609 (2h14m14.157213996s) [noise=true , address=77.90.31.13:4629 , piece_id=YCL43MLCDT7H5JEXZTJPFZPBYPCQDU5OBCZUDF6BWVCL6WH7D3WA, seg_pos={0 0}, upload=2055]: Done uploading piece. err: ecclient: upload failed (node:12GZtxLCqqDT4EXE2oCWjynZwboY9wBq4BtbkisCeTdCcCncT9f, address:77.90.31.13:4629): protocol: storage node overloaded, request limit: 2000 Jul 27 00:05:58.421789149 (2h14m14.20272253s) [upload=2055]: Next returned nil segment Jul 27 00:05:58.421812577 (2h14m14.202745964s) [upload=2055]: Waiting for error group managing segments... Jul 27 00:05:58.475786404 (2h14m14.256719782s) [noise=true , address=77.90.31.21:4896 , piece_id=C4KLG3F3ZFKGDLCVQ5TYMLLDY435RYGFKH3MRBMADOHP7ZK33LXA, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:05:58.514804015 (2h14m14.295737384s) [noise=true , address=77.90.31.11:4581 , piece_id=L5FL6RZAKVEEVE3FKYVSKMMWSZUE4AGCSIDPIPBUVNXOJ4N2PTMA, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:05:58.909211035 (2h14m14.690144414s) [noise=true , address=79.127.205.234:4607 , piece_id=IBB23QLHILMWRMWRCF4525OXXHOAD5F2WBGACMEQFSAU2RQKKMLQ, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:05:58.927170040 (2h14m14.708103421s) [noise=true , address=79.127.205.246:4284 , piece_id=MOZVL2IO5BVCGGXMCEHBR44N75Q6X4UFGKYGX7B5PJHBIRSV3PIA, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:05:58.934110828 (2h14m14.715044202s) [noise=true , address=79.127.205.233:4624 , piece_id=MM2AMS7F65ACLP7K7T4GPSLWSCZRAFYPSEAEICMVBIH27WPQUAYA, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:05:58.955956305 (2h14m14.736889685s) [noise=true , address=79.127.205.235:4678 , piece_id=ROLUJUTRCRCPUCH2UDR6CNL75TGR6OK4VP5DLJ5YZBG3SMBVYOIA, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:05:58.999211680 (2h14m14.780145047s) [noise=true , address=79.127.205.245:4227 , piece_id=NMI7IN2I546RHHWPK3G6YGFII6EMDMLKC7KH5QQ6AXQ5PVXC4DIQ, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:05:59.158673902 (2h14m14.939607287s) [noise=true , address=79.127.220.101:4048 , piece_id=5JCBUFMQ26LZO3V63U7TDF6EIC3CVMNLH2F4CXAWIJEP2X3CBTNA, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:06:01.209634590 (2h14m16.990567973s) [noise=true , address=64.49.13.138:4488 , piece_id=CZBKAYLTRLOTMU2QA2Q5THD4YVFGQ6SGPFQCOCCWE6MZ55S42GZQ, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:06:01.966716465 (2h14m17.747649837s) [noise=true , address=79.127.220.101:4041 , piece_id=SNPFNKV2APLHHCZOO6NZFJ5GYLSDHA6PXZ7Q4OY7VSOFLJMLP5MA, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:06:13.932563845 (2h14m29.71349723s) [noise=true , address=79.127.201.215:4351 , piece_id=HWJEB3QRKBAYY77XPW43B5QLIGZO7IKROYY7BRLBJZ2WJI2UR62A, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:06:26.446548545 (2h14m42.227481913s) [noise=true , address=77.90.31.12:4599 , piece_id=VUO6TD6RHSC2AS4W35SJE76FVPHDUAJSFIKYU3O4PKOD3A2BEEQA, seg_pos={0 0}, upload=2055]: Done uploading piece. err: <nil> Jul 27 00:06:58.246628477 (2h15m14.027561863s) [noise=true , address=104.36.82.42:4807 , piece_id=QHVUNNT6BC5PUJTUF2GK3NZVUS3QHMEGHR5Q3HSKY33GCPU3XXFQ, seg_pos={0 0}, upload=2055]: Done uploading piece. err: ecclient: upload failed (node:1iTiX3X69T9rKBxe1MLwpJEJPZyXVATQBP9cDEnQe2riXMGZeZ, address:104.36.82.42:4807): protocol: rpc: dial tcp 104.36.82.42:4807: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:58.246730364 (2h15m14.02766375s) [noise=true , address=104.36.82.56:4263 , piece_id=FX7AFYY7VA6LXBQXQBB6K6QFVDM7SSOL2M4HJW7ZZS4JNKODWEYQ, seg_pos={0 0}, upload=2055]: Done uploading piece. err: ecclient: upload failed (node:12tLW4r8AQKLCbSWZoNUVi5v2YtF8NthUmX81TLQn8G8TotuByG, address:104.36.82.56:4263): protocol: rpc: dial tcp 104.36.82.56:4263: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:58.246918178 (2h15m14.027851559s) [noise=true , address=104.36.82.41:4792 , piece_id=3VOW6ULEXAMKY5XKUPN2M2OF5QOC7UKWLNO5MGCQRQCLXOKL7DIQ, seg_pos={0 0}, upload=2055]: Done uploading piece. err: ecclient: upload failed (node:12pynuFncf2qop1yNZvkk9a1NsWPrvpYvwUnwXVcvKUQgpHMC2k, address:104.36.82.41:4792): protocol: rpc: dial tcp 104.36.82.41:4792: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:58.246931972 (2h15m14.027865357s) [noise=true , address=104.36.82.46:4513 , piece_id=RFUY2S6J5LTKLMCPJ5XUALZKVRLTVGQUXB2N7DUUL6LNXT2LRRAQ, seg_pos={0 0}, upload=2055]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: write tcp 159.223.143.39:60616->104.36.82.46:4513: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:58.246983250 (2h15m14.027916638s) [noise=true , address=104.36.82.47:4518 , piece_id=GYTLS7JW2CE67UI6XMAZBHSFDGO4X2N7PHBVRFHUNJ6AWXRAG33A, seg_pos={0 0}, upload=2055]: Done uploading piece. err: ecclient: upload failed (node:17TZ9JLdCivFF46guhSTW6hcoUmTz5uo5wvLkDqC4BGmpm9eLb, address:104.36.82.47:4518): protocol: write tcp 159.223.143.39:42512->104.36.82.47:4518: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:06:58.247215080 (2h15m14.028148494s) [seg_pos={0 0}, upload=2055]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:06:58.272223521 (2h15m14.053156899s) [upload=2055]: Done uploading segments... Jul 27 00:06:58.272255127 (2h15m14.053188508s) [upload=2055]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.029620968s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 54 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX....] duration: 1.058837634s FAIL (2025-07-27 00:06:39 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tore close: conn closed: op Jul 27 00:02:32.049171611 (2h10m47.830104991s) [noise=true , address=104.36.82.8:4969 , piece_id=YQPI3JIM2P3PXQ3MDOKX6PAC3XXSXAGKZRGVVKFW6VPWDTO4ZEKA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:02:32.049192849 (2h10m47.830126243s) [seg_pos={0 0}, upload=1997]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:02:32.076855520 (2h10m47.857788885s) [upload=1997]: Done uploading segments... Jul 27 00:02:32.076879200 (2h10m47.857812564s) [upload=1997]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:02:56.936460143 (2h11m12.717393528s) [upload=2012]: Starting upload Jul 27 00:02:56.936557479 (2h11m12.717490863s) [upload=2012]: Done starting upload Jul 27 00:02:56.936647830 (2h11m12.717581213s) [upload=2012]: Uploading segments... Jul 27 00:02:56.937017175 (2h11m12.71795055s) [upload=2012]: Got next segment. Inline: false Jul 27 00:02:56.937075780 (2h11m12.71800916s) [upload=2012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:02:56.937082791 (2h11m12.71801617s) [upload=2012]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:02:56.970415476 (2h11m12.751348861s) [seg_pos={0 0}, upload=2012]: Begin upload segment... Jul 27 00:02:56.970488660 (2h11m12.751422026s) [seg_pos={0 0}, upload=2012]: Done begin upload segment. Jul 27 00:02:56.970499199 (2h11m12.751432579s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970507686 (2h11m12.751441073s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970999895 (2h11m12.75193328s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.971544420 (2h11m12.752477804s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974009791 (2h11m12.754943174s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974480217 (2h11m12.755413597s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974879774 (2h11m12.755813153s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975304158 (2h11m12.756237538s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975704939 (2h11m12.756638317s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976240365 (2h11m12.757173746s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976662963 (2h11m12.757596345s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977149854 (2h11m12.75808323s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977565879 (2h11m12.758499264s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977962142 (2h11m12.758895525s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978370394 (2h11m12.759303791s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978686570 (2h11m12.759619953s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:57.069515194 (2h11m12.850448562s) [upload=2012]: Next returned nil segment Jul 27 00:02:57.069531895 (2h11m12.850465271s) [upload=2012]: Waiting for error group managing segments... Jul 27 00:02:57.073615634 (2h11m12.854549001s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074286917 (2h11m12.855220285s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074641770 (2h11m12.855575146s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074773340 (2h11m12.855706716s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075293141 (2h11m12.856226509s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075311903 (2h11m12.856245271s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075673862 (2h11m12.856607227s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075747686 (2h11m12.856681054s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.139972458 (2h11m12.920905828s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.172759261 (2h11m12.953692623s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:58.458246223 (2h11m14.2391796s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:03:26.903729789 (2h11m42.684663172s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903813623 (2h11m42.684747001s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903844203 (2h11m42.684777573s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903891506 (2h11m42.684824887s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:35200->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903946088 (2h11m42.684879465s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903986605 (2h11m42.684919991s) [seg_pos={0 0}, upload=2012]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:03:26.932638927 (2h11m42.713572307s) [upload=2012]: Done uploading segments... Jul 27 00:03:26.932697222 (2h11m42.713630598s) [upload=2012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 1.058867959s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 53 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX..] duration: 484.575542ms FAIL (2025-07-27 00:06:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 484.609457ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [.......X...X....XX..............X.X........X...XX...........] duration: 42.776574641s FAIL (2025-07-27 00:06:08 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.776642484s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....XXX...XX.....X.XXX...............X..X......X.XX....X...] duration: 2.069217237s FAIL (2025-07-27 00:06:05 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.069252801s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 59 out of 60 is failed (threshold=3) [X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.037864554s FAIL (2025-07-27 00:05:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== SegmentBegin Jul 27 00:04:58.210205117 (2h13m13.991138509s) [seg_pos={0 0}, upload=2040]: Begin upload segment... Jul 27 00:04:58.210287779 (2h13m13.991221171s) [seg_pos={0 0}, upload=2040]: Done begin upload segment. Jul 27 00:04:58.210650742 (2h13m13.99158413s) [noise=true , address=104.36.82.41:4780 , piece_id=2SVTYDMLGEELWC4XH6TLASYMUMW6LXX4QIINTY4PI7MSJ6SW26FQ, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.210711134 (2h13m13.991644513s) [noise=true , address=79.127.205.245:4226 , piece_id=DTHBLIXNKHVZUFKIGO4KV7AMKYEE5GFT4CK7P7E7HID2D4SYJZGA, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.211869232 (2h13m13.992802622s) [noise=true , address=79.127.205.234:4603 , piece_id=BYY3L5USEL43NCLA3KBVXL5S4X6RKL3FPUAWRTZC5K5OINIJRDYQ, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.212969363 (2h13m13.993902755s) [noise=true , address=104.36.82.54:4166 , piece_id=OJEMT4YGML5DH2QAU4UWPTXG3BEA2PZBOKZTB5V3WPR5SK3SNLQA, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.213491372 (2h13m13.994424759s) [noise=true , address=79.127.201.215:4349 , piece_id=7M5Y5INLLV3UMNBDRRKW737IJE546BQTBJ2FXBRXPGEKXJ44ZOQQ, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.213698188 (2h13m13.994631581s) [noise=true , address=64.49.13.140:4508 , piece_id=OOMWZTH5VHSX26YNTY2OIAD4DNVYM4WVCIPDE2EYLUB4GPU3CPUA, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.213916501 (2h13m13.994849888s) [noise=true , address=104.36.82.6:4887 , piece_id=LHGUXVTS3VNN4UJAZN5ED7GUACCA4OFRHXTSEOEZGP3LJ5IMP5RA, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.214398346 (2h13m13.995331734s) [noise=true , address=104.36.82.47:4526 , piece_id=AWELNJMDXZU2LG35FEOU6DLK2ZH6KQBPOQYLUPKYGFPA54QR2TWA, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.214430650 (2h13m13.995364022s) [noise=true , address=79.127.201.214:4289 , piece_id=AYEEO6F6PDE3BZQZU5PCERK65AQ27SR5NCEHXDY2RENI5YSIVWUQ, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.214900805 (2h13m13.99583419s) [noise=true , address=79.127.219.38:4004 , piece_id=NLFZ7OY7KOIQJUGFV27SNXDENPQJZXQW3HLE2S26SBZX7RS72NVQ, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.215513764 (2h13m13.996447153s) [noise=true , address=45.91.116.10:4093 , piece_id=DC22SLYAVXFR7N5RAAWXBZ3MY4TIYSSLCYLF3YOUWUC6V55XB6WA, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.215962482 (2h13m13.996895861s) [noise=true , address=104.36.82.8:4001 , piece_id=4ORP7YRXKVDRBY6XSLWFWGZZ7E7KGO6LU3APKQXXIUBHMBHSHTVQ, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.216023073 (2h13m13.996956464s) [noise=true , address=79.127.205.246:4288 , piece_id=DDSLWTUZX26JF6OJJMRU63AW5ZYJJNNBV4HBUESAT7TO4IN27LSA, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.216651063 (2h13m13.997584452s) [noise=true , address=79.127.205.235:4675 , piece_id=3DQRQ3LPZLD32VEAYWJS2K7KVG53M3O2OXHXLXKB423Q4X6WP7NA, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.217179857 (2h13m13.998113247s) [noise=true , address=77.90.31.21:4880 , piece_id=SK3YBP32VTDVNJMRWPN3A5JLWVG5OMHUI54KRJ5FDEGVYMCX6MYA, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.218088634 (2h13m13.999022026s) [noise=true , address=79.127.219.47:4048 , piece_id=DNKSVGGE2TYRPIVA5TE2BW5PG2S64UW7XS6BLLAJD6KGX7GD63EQ, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.218655413 (2h13m13.99958879s) [noise=true , address=77.90.31.13:4636 , piece_id=SAZB4S7UXMME7YIHVAABYYMLJYZFUWDXYKXXIFP7JLZGU2B5H22Q, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.218694842 (2h13m13.999628229s) [noise=true , address=104.36.82.52:4094 , piece_id=TOGFZPIGG7JSRP2ZTGDDZ7KC3HJSM22ERNPN6JDOH7NRBIULRSDQ, seg_pos={0 0}, upload=2040]: Uploading piece... Jul 27 00:04:58.315897196 (2h13m14.096830584s) [upload=2040]: Next returned nil segment Jul 27 00:04:58.315922828 (2h13m14.096856216s) [upload=2040]: Waiting for error group managing segments... Jul 27 00:04:58.344839771 (2h13m14.125773148s) [noise=true , address=79.127.219.38:4004 , piece_id=NLFZ7OY7KOIQJUGFV27SNXDENPQJZXQW3HLE2S26SBZX7RS72NVQ, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:04:58.361300725 (2h13m14.142234087s) [noise=true , address=77.90.31.21:4880 , piece_id=SK3YBP32VTDVNJMRWPN3A5JLWVG5OMHUI54KRJ5FDEGVYMCX6MYA, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:04:58.808295224 (2h13m14.589228599s) [noise=true , address=79.127.201.215:4349 , piece_id=7M5Y5INLLV3UMNBDRRKW737IJE546BQTBJ2FXBRXPGEKXJ44ZOQQ, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:04:58.811673220 (2h13m14.592606595s) [noise=true , address=79.127.205.245:4226 , piece_id=DTHBLIXNKHVZUFKIGO4KV7AMKYEE5GFT4CK7P7E7HID2D4SYJZGA, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:04:58.845410835 (2h13m14.626344204s) [noise=true , address=79.127.205.234:4603 , piece_id=BYY3L5USEL43NCLA3KBVXL5S4X6RKL3FPUAWRTZC5K5OINIJRDYQ, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:04:58.868675076 (2h13m14.649608445s) [noise=true , address=79.127.205.246:4288 , piece_id=DDSLWTUZX26JF6OJJMRU63AW5ZYJJNNBV4HBUESAT7TO4IN27LSA, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:04:58.869733797 (2h13m14.650667167s) [noise=true , address=79.127.205.235:4675 , piece_id=3DQRQ3LPZLD32VEAYWJS2K7KVG53M3O2OXHXLXKB423Q4X6WP7NA, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:04:59.073906603 (2h13m14.85483998s) [noise=true , address=79.127.219.47:4048 , piece_id=DNKSVGGE2TYRPIVA5TE2BW5PG2S64UW7XS6BLLAJD6KGX7GD63EQ, seg_pos={0 0}, upload=2040]: Done uploading piece. err: ecclient: upload failed (node:12LWLin5hV2KsxHKkUPMx7p3PDcFGb15CJsvbaGdjSy6wwcgRZL, address:79.127.219.47:4048): protocol: storage node overloaded, request limit: 2000 Jul 27 00:05:01.603186112 (2h13m17.384119495s) [noise=true , address=79.127.201.214:4289 , piece_id=AYEEO6F6PDE3BZQZU5PCERK65AQ27SR5NCEHXDY2RENI5YSIVWUQ, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:05:01.656202627 (2h13m17.437136008s) [noise=true , address=77.90.31.13:4636 , piece_id=SAZB4S7UXMME7YIHVAABYYMLJYZFUWDXYKXXIFP7JLZGU2B5H22Q, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:05:07.987842829 (2h13m23.768776207s) [noise=true , address=45.91.116.10:4093 , piece_id=DC22SLYAVXFR7N5RAAWXBZ3MY4TIYSSLCYLF3YOUWUC6V55XB6WA, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:05:09.863488221 (2h13m25.644421585s) [noise=true , address=64.49.13.140:4508 , piece_id=OOMWZTH5VHSX26YNTY2OIAD4DNVYM4WVCIPDE2EYLUB4GPU3CPUA, seg_pos={0 0}, upload=2040]: Done uploading piece. err: <nil> Jul 27 00:05:58.136527840 (2h14m13.917461217s) [noise=true , address=104.36.82.8:4001 , piece_id=4ORP7YRXKVDRBY6XSLWFWGZZ7E7KGO6LU3APKQXXIUBHMBHSHTVQ, seg_pos={0 0}, upload=2040]: Done uploading piece. err: ecclient: upload failed (node:12Eaef5RrMHiZS6dE7wTXyq239F1RXovU7UCTQ7uC6Qusj6bJDw, address:104.36.82.8:4001): protocol: rpc: dial tcp 104.36.82.8:4001: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:58.136601036 (2h14m13.917534405s) [noise=true , address=104.36.82.41:4780 , piece_id=2SVTYDMLGEELWC4XH6TLASYMUMW6LXX4QIINTY4PI7MSJ6SW26FQ, seg_pos={0 0}, upload=2040]: Done uploading piece. err: ecclient: upload failed (node:125WCbZ6a2NaewiLAHhHuPo91sQEfC9CZzEBV6rwF1JY7z13AEY, address:104.36.82.41:4780): protocol: write tcp 159.223.143.39:55260->104.36.82.41:4780: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:58.136647311 (2h14m13.917580676s) [noise=true , address=104.36.82.6:4887 , piece_id=LHGUXVTS3VNN4UJAZN5ED7GUACCA4OFRHXTSEOEZGP3LJ5IMP5RA, seg_pos={0 0}, upload=2040]: Done uploading piece. err: ecclient: upload failed (node:19QrU8gBXbSzy9LzhkHTYgdtTbZG51wV9CSWRcvvQuM6Jex9d7, address:104.36.82.6:4887): protocol: write tcp 159.223.143.39:38140->104.36.82.6:4887: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:58.136796710 (2h14m13.917730089s) [noise=true , address=104.36.82.54:4166 , piece_id=OJEMT4YGML5DH2QAU4UWPTXG3BEA2PZBOKZTB5V3WPR5SK3SNLQA, seg_pos={0 0}, upload=2040]: Done uploading piece. err: ecclient: upload failed (node:1BV7mWGAwKMXzmZBc3qAQFUXYyWtgQC7QEfG54YJgyJ4U3tFiv, address:104.36.82.54:4166): protocol: write tcp 159.223.143.39:40486->104.36.82.54:4166: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:58.136914293 (2h14m13.917847658s) [noise=true , address=104.36.82.47:4526 , piece_id=AWELNJMDXZU2LG35FEOU6DLK2ZH6KQBPOQYLUPKYGFPA54QR2TWA, seg_pos={0 0}, upload=2040]: Done uploading piece. err: ecclient: upload failed (node:1djnDo1bR9DZLEAHGJGoaxg895sQ3tQpYmPgGXpmsAxnpAtQwk, address:104.36.82.47:4526): protocol: rpc: dial tcp 104.36.82.47:4526: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:58.136970440 (2h14m13.917903816s) [noise=true , address=104.36.82.52:4094 , piece_id=TOGFZPIGG7JSRP2ZTGDDZ7KC3HJSM22ERNPN6JDOH7NRBIULRSDQ, seg_pos={0 0}, upload=2040]: Done uploading piece. err: ecclient: upload failed (node:1QYpeobqZ5QtLcuNBh9fHhiKxnbEtpiAaagvgWyTZvGbgbFxH7, address:104.36.82.52:4094): protocol: write tcp 159.223.143.39:33056->104.36.82.52:4094: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:05:58.137019619 (2h14m13.917953002s) [seg_pos={0 0}, upload=2040]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 27 00:05:58.171599389 (2h14m13.952532762s) [upload=2040]: Done uploading segments... Jul 27 00:05:58.171630335 (2h14m13.952563721s) [upload=2040]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.037888634s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX...] duration: 507.313538ms FAIL (2025-07-27 00:05:57 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tore close: conn closed: op Jul 27 00:02:32.049171611 (2h10m47.830104991s) [noise=true , address=104.36.82.8:4969 , piece_id=YQPI3JIM2P3PXQ3MDOKX6PAC3XXSXAGKZRGVVKFW6VPWDTO4ZEKA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:02:32.049192849 (2h10m47.830126243s) [seg_pos={0 0}, upload=1997]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:02:32.076855520 (2h10m47.857788885s) [upload=1997]: Done uploading segments... Jul 27 00:02:32.076879200 (2h10m47.857812564s) [upload=1997]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:02:56.936460143 (2h11m12.717393528s) [upload=2012]: Starting upload Jul 27 00:02:56.936557479 (2h11m12.717490863s) [upload=2012]: Done starting upload Jul 27 00:02:56.936647830 (2h11m12.717581213s) [upload=2012]: Uploading segments... Jul 27 00:02:56.937017175 (2h11m12.71795055s) [upload=2012]: Got next segment. Inline: false Jul 27 00:02:56.937075780 (2h11m12.71800916s) [upload=2012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:02:56.937082791 (2h11m12.71801617s) [upload=2012]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:02:56.970415476 (2h11m12.751348861s) [seg_pos={0 0}, upload=2012]: Begin upload segment... Jul 27 00:02:56.970488660 (2h11m12.751422026s) [seg_pos={0 0}, upload=2012]: Done begin upload segment. Jul 27 00:02:56.970499199 (2h11m12.751432579s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970507686 (2h11m12.751441073s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970999895 (2h11m12.75193328s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.971544420 (2h11m12.752477804s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974009791 (2h11m12.754943174s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974480217 (2h11m12.755413597s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974879774 (2h11m12.755813153s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975304158 (2h11m12.756237538s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975704939 (2h11m12.756638317s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976240365 (2h11m12.757173746s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976662963 (2h11m12.757596345s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977149854 (2h11m12.75808323s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977565879 (2h11m12.758499264s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977962142 (2h11m12.758895525s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978370394 (2h11m12.759303791s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978686570 (2h11m12.759619953s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:57.069515194 (2h11m12.850448562s) [upload=2012]: Next returned nil segment Jul 27 00:02:57.069531895 (2h11m12.850465271s) [upload=2012]: Waiting for error group managing segments... Jul 27 00:02:57.073615634 (2h11m12.854549001s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074286917 (2h11m12.855220285s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074641770 (2h11m12.855575146s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074773340 (2h11m12.855706716s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075293141 (2h11m12.856226509s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075311903 (2h11m12.856245271s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075673862 (2h11m12.856607227s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075747686 (2h11m12.856681054s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.139972458 (2h11m12.920905828s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.172759261 (2h11m12.953692623s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:58.458246223 (2h11m14.2391796s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:03:26.903729789 (2h11m42.684663172s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903813623 (2h11m42.684747001s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903844203 (2h11m42.684777573s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903891506 (2h11m42.684824887s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:35200->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903946088 (2h11m42.684879465s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903986605 (2h11m42.684919991s) [seg_pos={0 0}, upload=2012]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:03:26.932638927 (2h11m42.713572307s) [upload=2012]: Done uploading segments... Jul 27 00:03:26.932697222 (2h11m42.713630598s) [upload=2012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 507.343223ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 54 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX.] duration: 1.809832875s FAIL (2025-07-27 00:05:48 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.809865677s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXXX] duration: 30.061581937s FAIL (2025-07-27 00:05:46 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.061609324s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXXX] duration: 30.029981076s FAIL (2025-07-27 00:05:11 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.030025148s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.027872642s FAIL (2025-07-27 00:04:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 256 (2h12m13.851182632s) [upload=2025]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:03:58.114046876 (2h12m13.894980387s) [seg_pos={0 0}, upload=2025]: Begin upload segment... Jul 27 00:03:58.114113103 (2h12m13.895046484s) [seg_pos={0 0}, upload=2025]: Done begin upload segment. Jul 27 00:03:58.114152307 (2h12m13.895085693s) [noise=true , address=64.49.13.154:4927 , piece_id=3M4S26SJQTYS5WYS5B6KF6LGTYBXRHS2J6AHI26BOP4K5M22YXLQ, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.129874143 (2h12m13.910807531s) [noise=true , address=104.36.82.54:4170 , piece_id=BVFZN6P2XD6AEA34MZKT4MN7PPGVUHUEDT6RQFVPOIMHCL4NUGZQ, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.130394360 (2h12m13.911327748s) [noise=true , address=77.90.31.13:4636 , piece_id=EMBHHFZ3NKBGCO5E77DTO7QFR73CHRWLOYHZRAJJNRJSXYJBNIWA, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.130997070 (2h12m13.911930457s) [noise=true , address=104.36.82.47:4549 , piece_id=6UDTTFYCYC5LBH5G4WM54UU6YRG2U4B65LONOHMOK3VIPKUTM7PQ, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.131588284 (2h12m13.912521671s) [noise=true , address=104.36.82.46:4496 , piece_id=AB4YCF2ADLAAR6IA7NWUWEMG22RWISWYVTMF5P6F22LY74VXGU6A, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.132105816 (2h12m13.913039179s) [noise=true , address=77.90.31.16:4736 , piece_id=57P5D4LLH4L6DCMTDRSOX2BMWG6JQCTKC352SFWNC7RLMLLKLNAQ, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.132570113 (2h12m13.913503496s) [noise=true , address=79.127.201.215:4354 , piece_id=YEIJFT75CAWLYWRFAXYYD6O6A5VT6QFKMZS6FZRVCUFDO66MX5ZQ, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.133069101 (2h12m13.91400249s) [noise=true , address=104.36.82.52:4111 , piece_id=QRZ6PRVLMEXUECFWJRINPT6NQJ2ZCYPSNPG4TXSPPKGZLF6HPTXA, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.133720330 (2h12m13.914653718s) [noise=true , address=104.36.82.42:4821 , piece_id=OIKULIQVGAQCWW4YGSTQLZ2HHPAEUSNGXCGZVBMXCLG4USNZ4QFQ, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.134209015 (2h12m13.915142406s) [noise=true , address=77.90.31.19:4871 , piece_id=S4OA45TJCK4L65QSKXHO6ZKQLDZZHHO7CINEQ7H52J5GGENG7NCQ, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.135012129 (2h12m13.915945519s) [noise=true , address=45.91.116.13:4157 , piece_id=DMB4A2Z7FJBM3MHJLNOKPGOWGKHRKI2ISEVHE45LI5JJQ3GLFVCA, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.136811748 (2h12m13.917745121s) [noise=true , address=64.49.13.138:4482 , piece_id=S66BKIBJVE3O3XJJR6LVQUEWDDXYUMAIFG6C5NMDM5NWIVIAI65Q, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.137262345 (2h12m13.918195733s) [noise=true , address=64.49.13.137:4502 , piece_id=U66CNXLCTNMYSJPRHTMOCOEXPEGP6RJFBRKGDUSBHLVBPTBV47QQ, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.137965913 (2h12m13.91889931s) [noise=true , address=79.127.205.246:4282 , piece_id=P5OO4M32L2HTZTVKMDQ4F4MGXLXKPZI6PGL5CWWXS3ZYAOQOLRGA, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.138576563 (2h12m13.919509954s) [noise=true , address=77.90.31.11:4574 , piece_id=ZYU43ARUUGC3HIVX7ADQP7CCLHNMLNU4BVCJPR6KHKW6DTYDPIFA, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.139293589 (2h12m13.920226972s) [noise=true , address=79.127.219.38:4017 , piece_id=KNHHFET6O43QARQBBPIYLOJWAWJ7AXF6KHJPHWNGDTLJ5HEUJLZA, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.139785639 (2h12m13.920719027s) [noise=true , address=77.90.31.21:4907 , piece_id=WN6EVB7WRX3KHDPYMP6D24Q3PWA3EKZCVHYSZ74RFDI3Y3Y7MYGA, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.140380986 (2h12m13.921314373s) [noise=true , address=104.36.82.54:4175 , piece_id=BEUCXUYBV7BI6LZVUXDRFKYVAUSUQJ472LNQPE3C4K5VRBNWE5QA, seg_pos={0 0}, upload=2025]: Uploading piece... Jul 27 00:03:58.271977562 (2h12m14.05291094s) [upload=2025]: Next returned nil segment Jul 27 00:03:58.271994972 (2h12m14.052928349s) [upload=2025]: Waiting for error group managing segments... Jul 27 00:03:58.318044729 (2h12m14.098978112s) [noise=true , address=77.90.31.11:4574 , piece_id=ZYU43ARUUGC3HIVX7ADQP7CCLHNMLNU4BVCJPR6KHKW6DTYDPIFA, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:03:58.339680307 (2h12m14.120613683s) [noise=true , address=77.90.31.21:4907 , piece_id=WN6EVB7WRX3KHDPYMP6D24Q3PWA3EKZCVHYSZ74RFDI3Y3Y7MYGA, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:03:59.017719650 (2h12m14.798653016s) [noise=true , address=79.127.219.38:4017 , piece_id=KNHHFET6O43QARQBBPIYLOJWAWJ7AXF6KHJPHWNGDTLJ5HEUJLZA, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:03:59.022755319 (2h12m14.803688681s) [noise=true , address=79.127.205.246:4282 , piece_id=P5OO4M32L2HTZTVKMDQ4F4MGXLXKPZI6PGL5CWWXS3ZYAOQOLRGA, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:04:00.477236483 (2h12m16.258169845s) [noise=true , address=77.90.31.13:4636 , piece_id=EMBHHFZ3NKBGCO5E77DTO7QFR73CHRWLOYHZRAJJNRJSXYJBNIWA, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:04:01.245636107 (2h12m17.02656947s) [noise=true , address=45.91.116.13:4157 , piece_id=DMB4A2Z7FJBM3MHJLNOKPGOWGKHRKI2ISEVHE45LI5JJQ3GLFVCA, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:04:02.143012772 (2h12m17.923946152s) [noise=true , address=77.90.31.19:4871 , piece_id=S4OA45TJCK4L65QSKXHO6ZKQLDZZHHO7CINEQ7H52J5GGENG7NCQ, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:04:03.867893306 (2h12m19.648826683s) [noise=true , address=64.49.13.138:4482 , piece_id=S66BKIBJVE3O3XJJR6LVQUEWDDXYUMAIFG6C5NMDM5NWIVIAI65Q, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:04:05.225529650 (2h12m21.006463025s) [noise=true , address=64.49.13.137:4502 , piece_id=U66CNXLCTNMYSJPRHTMOCOEXPEGP6RJFBRKGDUSBHLVBPTBV47QQ, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:04:07.858858960 (2h12m23.639792332s) [noise=true , address=64.49.13.154:4927 , piece_id=3M4S26SJQTYS5WYS5B6KF6LGTYBXRHS2J6AHI26BOP4K5M22YXLQ, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:04:14.409102513 (2h12m30.190035875s) [noise=true , address=79.127.201.215:4354 , piece_id=YEIJFT75CAWLYWRFAXYYD6O6A5VT6QFKMZS6FZRVCUFDO66MX5ZQ, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:04:14.650696276 (2h12m30.431629641s) [noise=true , address=77.90.31.16:4736 , piece_id=57P5D4LLH4L6DCMTDRSOX2BMWG6JQCTKC352SFWNC7RLMLLKLNAQ, seg_pos={0 0}, upload=2025]: Done uploading piece. err: <nil> Jul 27 00:04:58.028785784 (2h13m13.80971916s) [noise=true , address=104.36.82.54:4170 , piece_id=BVFZN6P2XD6AEA34MZKT4MN7PPGVUHUEDT6RQFVPOIMHCL4NUGZQ, seg_pos={0 0}, upload=2025]: Done uploading piece. err: ecclient: upload failed (node:1mcYgUVVeD2xo4pDhR7ywPLRgyA7GgejeErtLiWiPTV2AhbBRX, address:104.36.82.54:4170): protocol: write tcp 159.223.143.39:35264->104.36.82.54:4170: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:58.028868510 (2h13m13.809801875s) [noise=true , address=104.36.82.54:4175 , piece_id=BEUCXUYBV7BI6LZVUXDRFKYVAUSUQJ472LNQPE3C4K5VRBNWE5QA, seg_pos={0 0}, upload=2025]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: write tcp 159.223.143.39:34108->104.36.82.54:4175: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:58.029044515 (2h13m13.809977888s) [noise=true , address=104.36.82.46:4496 , piece_id=AB4YCF2ADLAAR6IA7NWUWEMG22RWISWYVTMF5P6F22LY74VXGU6A, seg_pos={0 0}, upload=2025]: Done uploading piece. err: ecclient: upload failed (node:12V8XhSWkyGUVYFR8MykifFAP1YAfGCwzqucDcMMfzw3xsVW77Z, address:104.36.82.46:4496): protocol: write tcp 159.223.143.39:36988->104.36.82.46:4496: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:58.029321639 (2h13m13.810255039s) [noise=true , address=104.36.82.42:4821 , piece_id=OIKULIQVGAQCWW4YGSTQLZ2HHPAEUSNGXCGZVBMXCLG4USNZ4QFQ, seg_pos={0 0}, upload=2025]: Done uploading piece. err: ecclient: upload failed (node:1Q2vaBr3JNY3jv8jVUTg9uzfWUtNjrd5c5Z9Yfi54D3JMRM1af, address:104.36.82.42:4821): protocol: write tcp 159.223.143.39:58022->104.36.82.42:4821: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:58.029356741 (2h13m13.810290107s) [noise=true , address=104.36.82.52:4111 , piece_id=QRZ6PRVLMEXUECFWJRINPT6NQJ2ZCYPSNPG4TXSPPKGZLF6HPTXA, seg_pos={0 0}, upload=2025]: Done uploading piece. err: ecclient: upload failed (node:12krqZcFoBPfLFbLGQUAjPMzHJ4HKqsKPRg6Xap7nuShaEKBumH, address:104.36.82.52:4111): protocol: write tcp 159.223.143.39:39766->104.36.82.52:4111: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:58.029396743 (2h13m13.810330112s) [noise=true , address=104.36.82.47:4549 , piece_id=6UDTTFYCYC5LBH5G4WM54UU6YRG2U4B65LONOHMOK3VIPKUTM7PQ, seg_pos={0 0}, upload=2025]: Done uploading piece. err: ecclient: upload failed (node:1HWoyMKQ6v2XWUfEmfFh3m8AiGivAcYkkZ9na8kqHFzH2hPDeK, address:104.36.82.47:4549): protocol: write tcp 159.223.143.39:55352->104.36.82.47:4549: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:04:58.029437165 (2h13m13.810370556s) [seg_pos={0 0}, upload=2025]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 27 00:04:58.053486151 (2h13m13.834419518s) [upload=2025]: Done uploading segments... Jul 27 00:04:58.053533995 (2h13m13.834467361s) [upload=2025]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration:* 1m0.027894959s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [X.......X...X....XX..............X.X........X...XX..........] duration: 33.281110891s FAIL (2025-07-27 00:04:58 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 33.281131799s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......XXX...XX.....X.XXX...............X..X......X.XX....X..] duration: 1.860637841s FAIL (2025-07-27 00:04:51 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.860942313s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX..] duration: 2.010817168s FAIL (2025-07-27 00:04:48 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tore close: conn closed: op Jul 27 00:02:32.049171611 (2h10m47.830104991s) [noise=true , address=104.36.82.8:4969 , piece_id=YQPI3JIM2P3PXQ3MDOKX6PAC3XXSXAGKZRGVVKFW6VPWDTO4ZEKA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:02:32.049192849 (2h10m47.830126243s) [seg_pos={0 0}, upload=1997]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:02:32.076855520 (2h10m47.857788885s) [upload=1997]: Done uploading segments... Jul 27 00:02:32.076879200 (2h10m47.857812564s) [upload=1997]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:02:56.936460143 (2h11m12.717393528s) [upload=2012]: Starting upload Jul 27 00:02:56.936557479 (2h11m12.717490863s) [upload=2012]: Done starting upload Jul 27 00:02:56.936647830 (2h11m12.717581213s) [upload=2012]: Uploading segments... Jul 27 00:02:56.937017175 (2h11m12.71795055s) [upload=2012]: Got next segment. Inline: false Jul 27 00:02:56.937075780 (2h11m12.71800916s) [upload=2012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:02:56.937082791 (2h11m12.71801617s) [upload=2012]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:02:56.970415476 (2h11m12.751348861s) [seg_pos={0 0}, upload=2012]: Begin upload segment... Jul 27 00:02:56.970488660 (2h11m12.751422026s) [seg_pos={0 0}, upload=2012]: Done begin upload segment. Jul 27 00:02:56.970499199 (2h11m12.751432579s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970507686 (2h11m12.751441073s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970999895 (2h11m12.75193328s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.971544420 (2h11m12.752477804s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974009791 (2h11m12.754943174s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974480217 (2h11m12.755413597s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974879774 (2h11m12.755813153s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975304158 (2h11m12.756237538s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975704939 (2h11m12.756638317s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976240365 (2h11m12.757173746s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976662963 (2h11m12.757596345s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977149854 (2h11m12.75808323s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977565879 (2h11m12.758499264s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977962142 (2h11m12.758895525s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978370394 (2h11m12.759303791s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978686570 (2h11m12.759619953s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:57.069515194 (2h11m12.850448562s) [upload=2012]: Next returned nil segment Jul 27 00:02:57.069531895 (2h11m12.850465271s) [upload=2012]: Waiting for error group managing segments... Jul 27 00:02:57.073615634 (2h11m12.854549001s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074286917 (2h11m12.855220285s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074641770 (2h11m12.855575146s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074773340 (2h11m12.855706716s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075293141 (2h11m12.856226509s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075311903 (2h11m12.856245271s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075673862 (2h11m12.856607227s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075747686 (2h11m12.856681054s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.139972458 (2h11m12.920905828s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.172759261 (2h11m12.953692623s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:58.458246223 (2h11m14.2391796s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:03:26.903729789 (2h11m42.684663172s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903813623 (2h11m42.684747001s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903844203 (2h11m42.684777573s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903891506 (2h11m42.684824887s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:35200->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903946088 (2h11m42.684879465s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903986605 (2h11m42.684919991s) [seg_pos={0 0}, upload=2012]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:03:26.932638927 (2h11m42.713572307s) [upload=2012]: Done uploading segments... Jul 27 00:03:26.932697222 (2h11m42.713630598s) [upload=2012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 2.010851346s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [.X.......X...X....XX..............X.X........X...XX.........] duration: 29.661061768s FAIL (2025-07-27 00:04:25 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 29.661144258s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXXX] duration: 30.032421221s FAIL (2025-07-27 00:04:04 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.032448816s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [X.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.026825777s FAIL (2025-07-27 00:03:57 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 36.82.47:4520 , piece_id=XUBM7K4GBSNFCRI25KW5WZ3TB6JGRPJKDYD5WTHJAV7B4SALESIA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.017070783 (2h11m13.79800417s) [noise=true , address=79.127.219.38:4018 , piece_id=VLNZB5PH33GZTLQLVLQAQEPTTJSL5ERBN2BYKECAPRJHJUHI7TJA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.017538287 (2h11m13.798471665s) [noise=true , address=64.49.13.138:4483 , piece_id=Y44MF2GI4CXVIP3R7ZQYW2J5AB4ILO4AJNGJJSYLMPV3JY5VN5TA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.017549030 (2h11m13.798482418s) [noise=true , address=77.90.31.16:4748 , piece_id=ZDPCPBQHSBVWJFCZGEGUAANVZTGGCNMLWNRS3JDK4P3IE6H2FMYA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.017906052 (2h11m13.798839435s) [noise=true , address=79.127.205.235:4679 , piece_id=VZQFYJ5YH2UX6EQ5RODDKHE6QTYRM4O7AXNGH2WTAZ5CDW4G2P2Q, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.018004815 (2h11m13.798938211s) [noise=true , address=64.49.13.138:4482 , piece_id=YR3ZW2HGMKDKP32ZWO6KUK3DXF7VVIKDPXRW4XU6NEABNQK4ITCA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.018240986 (2h11m13.799174365s) [noise=true , address=104.36.82.52:4120 , piece_id=5AB6OGGGPSEDVO3PITQEAQYD7EJPQD2Q22QGT44UIYCSI2KJRCUA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.018593327 (2h11m13.799526714s) [noise=true , address=79.127.205.246:4269 , piece_id=NVAXJ4U4B6K3J46RYXANKMQLPB65CRIKI5M6RWSKTUS55EXJHFUQ, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.019074924 (2h11m13.800008307s) [noise=true , address=104.36.82.53:4148 , piece_id=NADXC75XGFLZPO5KVESTOAAGCDNTADGIK2ZURLKTNND7CNTGGLKA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.019518168 (2h11m13.800451553s) [noise=true , address=45.91.116.14:4206 , piece_id=PJLBYAJG3TH3YELB3VONBXEXNOB6FKXCIT4NG7VSEQ67KWQEK6LQ, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.020010537 (2h11m13.800943928s) [noise=true , address=104.36.82.54:4175 , piece_id=D3CRUPGTXPRDHVDUD3GOMHTT2DJC4WDI327CD7WNKMPDGJ5O5VIA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.020611094 (2h11m13.801544483s) [noise=true , address=79.127.201.215:4345 , piece_id=MTGMHK45SVSFEJX5EDKL3RLYRCFDLRHP2ZCMQZ5XF7LXAPBAONPQ, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.020998948 (2h11m13.801932338s) [noise=true , address=104.36.82.13:4211 , piece_id=2NO46AEDJK7Q3XPXIBWYZ2VENG345IRLGSBTVD7ZW4H7CTAZW7PQ, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.021489131 (2h11m13.802422521s) [noise=true , address=79.127.220.101:4048 , piece_id=3JTIL3DQRMYQMNGMRFIRNLWCZWXUQQMJQ2CWTLTSKO2ZNYVSBIQQ, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.021935419 (2h11m13.802868783s) [noise=true , address=104.36.82.20:4443 , piece_id=FMXBSLQRS5SG63CF5LHDTZMBTHZWEWBWHKELB7BU7NACFVVKZL4A, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.023066631 (2h11m13.804000023s) [noise=true , address=77.90.31.11:4579 , piece_id=R7ZSAQR7Z4VWDMTDJNVBIBDREEIHX5XEHMZ2I4OM2ENLNJD7GUBQ, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.023950660 (2h11m13.804884044s) [noise=true , address=104.36.82.46:4499 , piece_id=IGGQR5V36IQDZEXRIQW2NB4AH6XXMKAUUSTGONVUVMOGMG4LUSNA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.049779256 (2h11m13.830712646s) [noise=true , address=79.127.205.234:4618 , piece_id=EXLWSGG6FWHQRT53BDSECCVSQUG4BVSFYCDOWMR2CJTGAZK2ZDFA, seg_pos={0 0}, upload=2013]: Uploading piece... Jul 27 00:02:58.112187641 (2h11m13.89312102s) [upload=2013]: Next returned nil segment Jul 27 00:02:58.112207795 (2h11m13.893141182s) [upload=2013]: Waiting for error group managing segments... Jul 27 00:02:58.116592894 (2h11m13.897526281s) [noise=true , address=79.127.219.38:4018 , piece_id=VLNZB5PH33GZTLQLVLQAQEPTTJSL5ERBN2BYKECAPRJHJUHI7TJA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:02:58.154311610 (2h11m13.935244992s) [noise=true , address=77.90.31.11:4579 , piece_id=R7ZSAQR7Z4VWDMTDJNVBIBDREEIHX5XEHMZ2I4OM2ENLNJD7GUBQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:02:58.623605977 (2h11m14.404539339s) [noise=true , address=79.127.205.246:4269 , piece_id=NVAXJ4U4B6K3J46RYXANKMQLPB65CRIKI5M6RWSKTUS55EXJHFUQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:02:58.635272139 (2h11m14.416205507s) [noise=true , address=79.127.205.235:4679 , piece_id=VZQFYJ5YH2UX6EQ5RODDKHE6QTYRM4O7AXNGH2WTAZ5CDW4G2P2Q, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:02:58.680942641 (2h11m14.461876023s) [noise=true , address=79.127.205.234:4618 , piece_id=EXLWSGG6FWHQRT53BDSECCVSQUG4BVSFYCDOWMR2CJTGAZK2ZDFA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:02:59.109300394 (2h11m14.89023377s) [noise=true , address=79.127.220.101:4048 , piece_id=3JTIL3DQRMYQMNGMRFIRNLWCZWXUQQMJQ2CWTLTSKO2ZNYVSBIQQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:03:05.870854136 (2h11m21.651787504s) [noise=true , address=64.49.13.138:4483 , piece_id=Y44MF2GI4CXVIP3R7ZQYW2J5AB4ILO4AJNGJJSYLMPV3JY5VN5TA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:03:06.380469285 (2h11m22.161402667s) [noise=true , address=64.49.13.138:4482 , piece_id=YR3ZW2HGMKDKP32ZWO6KUK3DXF7VVIKDPXRW4XU6NEABNQK4ITCA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:03:08.016464682 (2h11m23.797398054s) [noise=true , address=45.91.116.14:4206 , piece_id=PJLBYAJG3TH3YELB3VONBXEXNOB6FKXCIT4NG7VSEQ67KWQEK6LQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:03:38.212561053 (2h11m53.993494435s) [noise=true , address=79.127.201.215:4345 , piece_id=MTGMHK45SVSFEJX5EDKL3RLYRCFDLRHP2ZCMQZ5XF7LXAPBAONPQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: <nil> Jul 27 00:03:57.938576320 (2h12m13.719509708s) [noise=true , address=77.90.31.16:4748 , piece_id=ZDPCPBQHSBVWJFCZGEGUAANVZTGGCNMLWNRS3JDK4P3IE6H2FMYA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: ecclient: upload failed (node:12k122BWfbKgBxSs8MCfNFkZYh2WfZDDHLktLQRiifkPr95G7SY, address:77.90.31.16:4748): protocol: rpc: dial tcp 77.90.31.16:4748: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:57.938729802 (2h12m13.719663182s) [noise=true , address=104.36.82.52:4120 , piece_id=5AB6OGGGPSEDVO3PITQEAQYD7EJPQD2Q22QGT44UIYCSI2KJRCUA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: ecclient: upload failed (node:1aKMhWsxNocDaqpuPLy7177jEA9K6MrsuKi1awcDcod3mM1vM3, address:104.36.82.52:4120): protocol: write tcp 159.223.143.39:59044->104.36.82.52:4120: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:57.938823589 (2h12m13.719756966s) [noise=true , address=104.36.82.54:4175 , piece_id=D3CRUPGTXPRDHVDUD3GOMHTT2DJC4WDI327CD7WNKMPDGJ5O5VIA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: rpc: dial tcp 104.36.82.54:4175: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:57.938834539 (2h12m13.719767908s) [noise=true , address=104.36.82.46:4499 , piece_id=IGGQR5V36IQDZEXRIQW2NB4AH6XXMKAUUSTGONVUVMOGMG4LUSNA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: ecclient: upload failed (node:1JR5bVTizvQCY5utaGUDgLHvCJfrbbpixmKNrq5vjiRB8VaSoc, address:104.36.82.46:4499): protocol: rpc: dial tcp 104.36.82.46:4499: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:57.938883165 (2h12m13.719816542s) [noise=true , address=104.36.82.47:4520 , piece_id=XUBM7K4GBSNFCRI25KW5WZ3TB6JGRPJKDYD5WTHJAV7B4SALESIA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: ecclient: upload failed (node:12eNRsboM5zFiyBVPLuQfgb5QSAc5bZFizK9KJHudNd9nstVGyy, address:104.36.82.47:4520): protocol: write tcp 159.223.143.39:40904->104.36.82.47:4520: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:57.938964526 (2h12m13.719897908s) [noise=true , address=104.36.82.53:4148 , piece_id=NADXC75XGFLZPO5KVESTOAAGCDNTADGIK2ZURLKTNND7CNTGGLKA, seg_pos={0 0}, upload=2013]: Done uploading piece. err: ecclient: upload failed (node:12GGwKp8TvH5JJ8mcR4sqKcGMCc56hMEgWpAR3zqTWerqTLQ9Rv, address:104.36.82.53:4148): protocol: rpc: dial tcp 104.36.82.53:4148: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:57.939008149 (2h12m13.719941527s) [noise=true , address=104.36.82.20:4443 , piece_id=FMXBSLQRS5SG63CF5LHDTZMBTHZWEWBWHKELB7BU7NACFVVKZL4A, seg_pos={0 0}, upload=2013]: Done uploading piece. err: ecclient: upload failed (node:1rTVUAPUj4PUy8qHuutnB2WYCVpJdiVwfutA6UK4TZoPzvQEUv, address:104.36.82.20:4443): protocol: write tcp 159.223.143.39:36038->104.36.82.20:4443: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:57.939074361 (2h12m13.720007738s) [noise=true , address=104.36.82.13:4211 , piece_id=2NO46AEDJK7Q3XPXIBWYZ2VENG345IRLGSBTVD7ZW4H7CTAZW7PQ, seg_pos={0 0}, upload=2013]: Done uploading piece. err: ecclient: upload failed (node:12dANoYTygpcpWfCcAJtsnVBFWSekLhDjheAXnsaYMChRYAm2qc, address:104.36.82.13:4211): protocol: write tcp 159.223.143.39:46864->104.36.82.13:4211: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:57.939111306 (2h12m13.720044706s) [seg_pos={0 0}, upload=2013]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:03:57.961726382 (2h12m13.742659746s) [upload=2013]: Done uploading segments... Jul 27 00:03:57.961758648 (2h12m13.742692024s) [upload=2013]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.02684651s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.......XXX...XX.....X.XXX...............X..X......X.XX....X.] duration: 652.205884ms FAIL (2025-07-27 00:03:40 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 652.226519ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.......XXX...XX.....X.XXX...............X..X......X.XX....X] duration: 30.00041246s FAIL (2025-07-27 00:03:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000435887s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 57 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX.] duration: 1.725330465s FAIL (2025-07-27 00:03:34 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tore close: conn closed: op Jul 27 00:02:32.049171611 (2h10m47.830104991s) [noise=true , address=104.36.82.8:4969 , piece_id=YQPI3JIM2P3PXQ3MDOKX6PAC3XXSXAGKZRGVVKFW6VPWDTO4ZEKA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:02:32.049192849 (2h10m47.830126243s) [seg_pos={0 0}, upload=1997]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:02:32.076855520 (2h10m47.857788885s) [upload=1997]: Done uploading segments... Jul 27 00:02:32.076879200 (2h10m47.857812564s) [upload=1997]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:02:56.936460143 (2h11m12.717393528s) [upload=2012]: Starting upload Jul 27 00:02:56.936557479 (2h11m12.717490863s) [upload=2012]: Done starting upload Jul 27 00:02:56.936647830 (2h11m12.717581213s) [upload=2012]: Uploading segments... Jul 27 00:02:56.937017175 (2h11m12.71795055s) [upload=2012]: Got next segment. Inline: false Jul 27 00:02:56.937075780 (2h11m12.71800916s) [upload=2012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:02:56.937082791 (2h11m12.71801617s) [upload=2012]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:02:56.970415476 (2h11m12.751348861s) [seg_pos={0 0}, upload=2012]: Begin upload segment... Jul 27 00:02:56.970488660 (2h11m12.751422026s) [seg_pos={0 0}, upload=2012]: Done begin upload segment. Jul 27 00:02:56.970499199 (2h11m12.751432579s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970507686 (2h11m12.751441073s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970999895 (2h11m12.75193328s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.971544420 (2h11m12.752477804s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974009791 (2h11m12.754943174s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974480217 (2h11m12.755413597s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974879774 (2h11m12.755813153s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975304158 (2h11m12.756237538s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975704939 (2h11m12.756638317s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976240365 (2h11m12.757173746s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976662963 (2h11m12.757596345s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977149854 (2h11m12.75808323s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977565879 (2h11m12.758499264s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977962142 (2h11m12.758895525s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978370394 (2h11m12.759303791s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978686570 (2h11m12.759619953s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:57.069515194 (2h11m12.850448562s) [upload=2012]: Next returned nil segment Jul 27 00:02:57.069531895 (2h11m12.850465271s) [upload=2012]: Waiting for error group managing segments... Jul 27 00:02:57.073615634 (2h11m12.854549001s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074286917 (2h11m12.855220285s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074641770 (2h11m12.855575146s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074773340 (2h11m12.855706716s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075293141 (2h11m12.856226509s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075311903 (2h11m12.856245271s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075673862 (2h11m12.856607227s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075747686 (2h11m12.856681054s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.139972458 (2h11m12.920905828s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.172759261 (2h11m12.953692623s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:58.458246223 (2h11m14.2391796s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:03:26.903729789 (2h11m42.684663172s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903813623 (2h11m42.684747001s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903844203 (2h11m42.684777573s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903891506 (2h11m42.684824887s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:35200->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903946088 (2h11m42.684879465s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903986605 (2h11m42.684919991s) [seg_pos={0 0}, upload=2012]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:03:26.932638927 (2h11m42.713572307s) [upload=2012]: Done uploading segments... Jul 27 00:03:26.932697222 (2h11m42.713630598s) [upload=2012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 1.725372538s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [..X.......X...X....XX..............X.X........X...XX........] duration: 36.095016896s FAIL (2025-07-27 00:03:32 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 36.095042213s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 30.03146759s FAIL (2025-07-27 00:03:26 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== tore close: conn closed: op Jul 27 00:02:32.049171611 (2h10m47.830104991s) [noise=true , address=104.36.82.8:4969 , piece_id=YQPI3JIM2P3PXQ3MDOKX6PAC3XXSXAGKZRGVVKFW6VPWDTO4ZEKA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:02:32.049192849 (2h10m47.830126243s) [seg_pos={0 0}, upload=1997]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:02:32.076855520 (2h10m47.857788885s) [upload=1997]: Done uploading segments... Jul 27 00:02:32.076879200 (2h10m47.857812564s) [upload=1997]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 27 00:02:56.936460143 (2h11m12.717393528s) [upload=2012]: Starting upload Jul 27 00:02:56.936557479 (2h11m12.717490863s) [upload=2012]: Done starting upload Jul 27 00:02:56.936647830 (2h11m12.717581213s) [upload=2012]: Uploading segments... Jul 27 00:02:56.937017175 (2h11m12.71795055s) [upload=2012]: Got next segment. Inline: false Jul 27 00:02:56.937075780 (2h11m12.71800916s) [upload=2012]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:02:56.937082791 (2h11m12.71801617s) [upload=2012]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:02:56.970415476 (2h11m12.751348861s) [seg_pos={0 0}, upload=2012]: Begin upload segment... Jul 27 00:02:56.970488660 (2h11m12.751422026s) [seg_pos={0 0}, upload=2012]: Done begin upload segment. Jul 27 00:02:56.970499199 (2h11m12.751432579s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970507686 (2h11m12.751441073s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.970999895 (2h11m12.75193328s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.971544420 (2h11m12.752477804s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974009791 (2h11m12.754943174s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974480217 (2h11m12.755413597s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.974879774 (2h11m12.755813153s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975304158 (2h11m12.756237538s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.975704939 (2h11m12.756638317s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976240365 (2h11m12.757173746s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.976662963 (2h11m12.757596345s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977149854 (2h11m12.75808323s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977565879 (2h11m12.758499264s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.977962142 (2h11m12.758895525s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978370394 (2h11m12.759303791s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:56.978686570 (2h11m12.759619953s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Uploading piece... Jul 27 00:02:57.069515194 (2h11m12.850448562s) [upload=2012]: Next returned nil segment Jul 27 00:02:57.069531895 (2h11m12.850465271s) [upload=2012]: Waiting for error group managing segments... Jul 27 00:02:57.073615634 (2h11m12.854549001s) [noise=true , address=45.91.116.11:4119 , piece_id=HL32I3YAFDLE2TWPLC3LCVEZHZYDUARVALNAOCGB6UOV7UFLB62Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074286917 (2h11m12.855220285s) [noise=true , address=45.91.116.10:4081 , piece_id=ADQPMPARABEVORG4H2T57TTYCDK7O765V5NCBUZWCKJEWLUX7CUA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074641770 (2h11m12.855575146s) [noise=true , address=45.91.116.12:4226 , piece_id=K3PNDUPG273MLRWT2YECKIE2UKA2AD4ULEJOTQ2JT57IH6CPRQ3Q, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.074773340 (2h11m12.855706716s) [noise=true , address=77.90.31.11:4553 , piece_id=BYC44AZ7WNGCIGMMRAO7HMLYQIL77MMW7QW4NML42RI5B7I2WBXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075293141 (2h11m12.856226509s) [noise=true , address=45.91.116.7:4011 , piece_id=57X7DWURFLOZ5R37PT4RWJVNBSWK3MZMF2EMBADVLYBOYE434HUQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075311903 (2h11m12.856245271s) [noise=true , address=45.91.116.8:4048 , piece_id=RESCJIPLELABKK25DLIDII4RMJOQH27APLVRYJXDMLSRCD5XZCKA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075673862 (2h11m12.856607227s) [noise=true , address=45.91.116.13:4154 , piece_id=SRRO75O4PHXEIWRMU4XPWJ2YNNWBLD4V4IXET6CHSIG4LSZEWHCQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.075747686 (2h11m12.856681054s) [noise=true , address=77.90.31.13:4626 , piece_id=ZUI32V43VX2XLTDP4PDU4CA67ZTY7QKFB7SZ57F6K7DAJRNRZUEQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.139972458 (2h11m12.920905828s) [noise=true , address=45.91.116.14:4189 , piece_id=X4UTSS7DOUIKS3A5JG7ZXBMD2HVTEFJKGVGYQEQER554JJKAZAHA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:57.172759261 (2h11m12.953692623s) [noise=true , address=45.91.116.14:4190 , piece_id=HCSJBS6TRITNMBMZVEPEMEY7W63TLGKAPLQ2YWFX32GK64RNWIGA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:02:58.458246223 (2h11m14.2391796s) [noise=true , address=77.90.31.12:4592 , piece_id=XWTJRZQZJBKH7Q4ODQLTAD4NBYAYGMU7GS4OMIOVVXQZVIJ4BTMA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: <nil> Jul 27 00:03:26.903729789 (2h11m42.684663172s) [noise=true , address=77.90.31.15:4699 , piece_id=K7ETJY7N7QEQ4QADIYFVXMXAOVZQSWP6HNK6U43XJM7HQ2HFGLXA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12Rb4DbV6syjrfdRBUUZQ4efA3xtwDFawUFp5GdQRc4GaDS2T8M, address:77.90.31.15:4699): protocol: rpc: dial tcp 77.90.31.15:4699: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903813623 (2h11m42.684747001s) [noise=true , address=104.36.82.10:4077 , piece_id=YDRUZX2BCRVFP74R5JF3I4FXGBWTZPEUJUUDRPKGGC2KUOIY4F5A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903844203 (2h11m42.684777573s) [noise=true , address=104.36.82.12:4150 , piece_id=57QCOKI2XVSEIJLORTFUPL6XGRSMLLQ6CVVMIGXERHOOBIJT3O6A, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903891506 (2h11m42.684824887s) [noise=true , address=104.36.82.11:4115 , piece_id=DBJBSFHH2GO42JTN5WMJZJIIKW6GCLQPWASIQHOI3BY7PXYQUZAA, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:35200->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903946088 (2h11m42.684879465s) [noise=true , address=104.36.82.9:4007 , piece_id=EPXVGAMUEL2EJ22MG2ADU7EDZF7DHZN4DJG7MH5HMFECOT2YK5IQ, seg_pos={0 0}, upload=2012]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:03:26.903986605 (2h11m42.684919991s) [seg_pos={0 0}, upload=2012]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:03:26.932638927 (2h11m42.713572307s) [upload=2012]: Done uploading segments... Jul 27 00:03:26.932697222 (2h11m42.713630598s) [upload=2012]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.031522112s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XXX] duration: 30.031885123s FAIL (2025-07-27 00:03:15 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.031910465s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [XX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.032562196s FAIL (2025-07-27 00:02:57 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== 3CZZYAGBID4AEN5ZYET6UGRX4CAPAQ, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.890168627 (2h10m13.671102013s) [noise=true , address=104.36.82.56:4254 , piece_id=CQFUNQLCXWVW3DAX57BLSKPSY7Y7ECR3VTNGEMEWSFYABBP7BJEA, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.890647405 (2h10m13.671580792s) [noise=true , address=64.49.13.140:4508 , piece_id=MOT2SIO4D3RR5EDLTY2YMYTSGAKEWOHRPFAK4JXX2NUEZKXORGOQ, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.892500235 (2h10m13.67343362s) [noise=true , address=45.91.116.9:4277 , piece_id=HJEOOMIBFL7KIW4GK3G4KHQTAWAPIRCO3KFM33P4VTMLKVBDDL7Q, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.893017271 (2h10m13.673950655s) [noise=true , address=77.90.31.18:4833 , piece_id=ZWGOHSBPQWSXIQGG6QUYAXNJUQGFRWDCRRYXTPICZHH5QEJZXD4A, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.893432369 (2h10m13.674365751s) [noise=true , address=104.36.82.53:4141 , piece_id=S2K2VNMRALLQCWETY3E4ZO7M2PLGI4E3GHYMPHXYSIFZ2PWREEZQ, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.893844336 (2h10m13.674777721s) [noise=true , address=79.127.205.235:4657 , piece_id=26BBTLP4HWQ4P7TLLUFBR7PPR6JZU6DX73WP5JLJ7SWFPKKUUMLA, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.894293428 (2h10m13.675226815s) [noise=true , address=77.90.31.17:4781 , piece_id=235LMHPJIKW422NJLWPFWML453IVPRSBQKHKHYUHGPDAQJJL3JVA, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.894714611 (2h10m13.67564799s) [noise=true , address=104.36.82.46:4498 , piece_id=2Q5GHPRMO6ZN6SMWAL7JT7NMPPHISX56XJXCFFK466PIXWBWBCLQ, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.895164078 (2h10m13.676097465s) [noise=true , address=77.90.31.15:4732 , piece_id=UCCKTYWVWENHC5DU6KAISHSAHQ6ZJPBYCVQF55PC3PG35RS3TYVQ, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.895566819 (2h10m13.676500206s) [noise=true , address=104.36.82.55:4218 , piece_id=AA75RMM7UZ45ZLGEGTBX6477MSBAJWPGIJ2QMHU2MERTHPMWUROA, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.895993151 (2h10m13.676926539s) [noise=true , address=79.127.201.215:4345 , piece_id=HLRWINXDAB7L7ILI22DU2E7DAYKKBF2NL254R6T4W4ABLVNXYCSQ, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.896414415 (2h10m13.6773478s) [noise=true , address=156.146.43.225:4449 , piece_id=BI4IEVSJU6DRU5CHVHLJ3GXYMUX42XXCQTXXDEBIMDVUGS5IU4HQ, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.899363215 (2h10m13.680296608s) [noise=true , address=104.36.82.47:4524 , piece_id=6SPF5NA2NNC6TPVHUPOMVYHLSHQDQ4NVG3FI5DCMHLNNRK6JUWZA, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.899741795 (2h10m13.680675177s) [noise=true , address=104.36.82.54:4167 , piece_id=WJFNYIJHDLI5Q43WOAEKFPCNH6DSSEF3FTPGNTZ6QZT4PMKPWJ5A, seg_pos={0 0}, upload=1995]: Uploading piece... Jul 27 00:01:57.968961593 (2h10m13.749894969s) [upload=1995]: Next returned nil segment Jul 27 00:01:57.968981228 (2h10m13.749914598s) [upload=1995]: Waiting for error group managing segments... Jul 27 00:01:58.000683759 (2h10m13.781617141s) [noise=true , address=77.90.31.18:4833 , piece_id=ZWGOHSBPQWSXIQGG6QUYAXNJUQGFRWDCRRYXTPICZHH5QEJZXD4A, seg_pos={0 0}, upload=1995]: Done uploading piece. err: <nil> Jul 27 00:01:58.041401918 (2h10m13.822335295s) [noise=true , address=156.146.43.225:4449 , piece_id=BI4IEVSJU6DRU5CHVHLJ3GXYMUX42XXCQTXXDEBIMDVUGS5IU4HQ, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:12sjnjDAH1FS9CNxSfYy3gBL2e19uQoyeWVeGtSFVcpRK9SYYZk, address:156.146.43.225:4449): protocol: storage node overloaded, request limit: 2000 Jul 27 00:01:58.472609921 (2h10m14.253543298s) [noise=true , address=79.127.205.234:4620 , piece_id=M4YJIHRO4GY4EATF3MLB75GQT2QOWUNIP2Y5S5QP5IO42X4NY26Q, seg_pos={0 0}, upload=1995]: Done uploading piece. err: <nil> Jul 27 00:01:59.085103370 (2h10m14.866036737s) [noise=true , address=79.127.205.235:4657 , piece_id=26BBTLP4HWQ4P7TLLUFBR7PPR6JZU6DX73WP5JLJ7SWFPKKUUMLA, seg_pos={0 0}, upload=1995]: Done uploading piece. err: <nil> Jul 27 00:02:00.633095575 (2h10m16.414028951s) [noise=true , address=79.127.220.97:4361 , piece_id=YXD5TXHRDT2M7U44FLXT6U3CZZYAGBID4AEN5ZYET6UGRX4CAPAQ, seg_pos={0 0}, upload=1995]: Done uploading piece. err: <nil> Jul 27 00:02:05.406614524 (2h10m21.187547892s) [noise=true , address=64.49.13.140:4508 , piece_id=MOT2SIO4D3RR5EDLTY2YMYTSGAKEWOHRPFAK4JXX2NUEZKXORGOQ, seg_pos={0 0}, upload=1995]: Done uploading piece. err: <nil> Jul 27 00:02:08.511226511 (2h10m24.292159889s) [noise=true , address=45.91.116.9:4277 , piece_id=HJEOOMIBFL7KIW4GK3G4KHQTAWAPIRCO3KFM33P4VTMLKVBDDL7Q, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:1wAuJKermCFTxgfjkq9sq54XDhXJvQN1FrM491irH1dFpDp51v, address:45.91.116.9:4277): protocol: storage node overloaded, request limit: 2000 Jul 27 00:02:25.830189629 (2h10m41.611122998s) [noise=true , address=79.127.201.215:4345 , piece_id=HLRWINXDAB7L7ILI22DU2E7DAYKKBF2NL254R6T4W4ABLVNXYCSQ, seg_pos={0 0}, upload=1995]: Done uploading piece. err: <nil> Jul 27 00:02:57.818009349 (2h11m13.598942728s) [noise=true , address=104.36.82.53:4141 , piece_id=S2K2VNMRALLQCWETY3E4ZO7M2PLGI4E3GHYMPHXYSIFZ2PWREEZQ, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:15wUF135h4QsyvF7Cr3eb6rxtx8KPz1LHbgRjQYwiF5xpmE2nn, address:104.36.82.53:4141): protocol: rpc: dial tcp 104.36.82.53:4141: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:57.818131163 (2h11m13.599064541s) [noise=true , address=77.90.31.17:4781 , piece_id=235LMHPJIKW422NJLWPFWML453IVPRSBQKHKHYUHGPDAQJJL3JVA, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:1Uc2Hn41mUYgLtyirRtEGNv5iKBv8YqzQo3RYdoE4M1Fj29WgT, address:77.90.31.17:4781): protocol: rpc: dial tcp 77.90.31.17:4781: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:57.818227960 (2h11m13.599161339s) [noise=true , address=104.36.82.55:4218 , piece_id=AA75RMM7UZ45ZLGEGTBX6477MSBAJWPGIJ2QMHU2MERTHPMWUROA, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:122z4Fys9DYGB3WnjvjiZ5TSA3h8dXvMFcaFfYdpMtxGT6ZiiGH, address:104.36.82.55:4218): protocol: rpc: dial tcp 104.36.82.55:4218: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:57.818303610 (2h11m13.599236987s) [noise=true , address=104.36.82.54:4167 , piece_id=WJFNYIJHDLI5Q43WOAEKFPCNH6DSSEF3FTPGNTZ6QZT4PMKPWJ5A, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:1hhT46NW5CQ7dWHRxy5MSjyKc3PiQqBifThT7dBmx2sgNvn8GZ, address:104.36.82.54:4167): protocol: rpc: dial tcp 104.36.82.54:4167: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:57.818418934 (2h11m13.599352329s) [noise=true , address=104.36.82.56:4254 , piece_id=CQFUNQLCXWVW3DAX57BLSKPSY7Y7ECR3VTNGEMEWSFYABBP7BJEA, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:1k4x3HL3PNHwKJX83RyqvnsATHk3HmeMiC12A7raLN58wzvkUN, address:104.36.82.56:4254): protocol: write tcp 159.223.143.39:35910->104.36.82.56:4254: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:57.818736110 (2h11m13.599669477s) [noise=true , address=104.36.82.35:4585 , piece_id=PEGX62F6H5HWLFCK6Z4MB3JL2452Z2EIMWWYNMQO54MEVKJAO6VQ, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:1TyK6onzVds1kucaD6Y4aPssPDM5FzzQ91dTrez5B7ttfwvXm, address:104.36.82.35:4585): protocol: rpc: dial tcp 104.36.82.35:4585: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:57.818826102 (2h11m13.599759481s) [noise=true , address=77.90.31.10:4523 , piece_id=FC4CXEWPO3RHMEP5ELANVLNZ22B4CVTMS5XOO5BLVFCRUOOYXUKA, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:1LifXs8irH3dbLrzPnXzoCfaFyuUJxsN4rfyLEwTKGXSPxotPa, address:77.90.31.10:4523): protocol: write tcp 159.223.143.39:52404->77.90.31.10:4523: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:57.818868442 (2h11m13.599801813s) [noise=true , address=104.36.82.46:4498 , piece_id=2Q5GHPRMO6ZN6SMWAL7JT7NMPPHISX56XJXCFFK466PIXWBWBCLQ, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:1jyjxdWkBRBkDHSKknNmPdcnJ3HRcSCUQNnmpb1CaiEdMPEGk4, address:104.36.82.46:4498): protocol: rpc: dial tcp 104.36.82.46:4498: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:57.819242911 (2h11m13.600176279s) [noise=true , address=104.36.82.47:4524 , piece_id=6SPF5NA2NNC6TPVHUPOMVYHLSHQDQ4NVG3FI5DCMHLNNRK6JUWZA, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:1xfbCzjSwFDRrR1gPggeN8tNCLbRRWDVzCJ6mqhZsbVSe8d4s4, address:104.36.82.47:4524): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:02:57.819329587 (2h11m13.600262957s) [noise=true , address=77.90.31.15:4732 , piece_id=UCCKTYWVWENHC5DU6KAISHSAHQ6ZJPBYCVQF55PC3PG35RS3TYVQ, seg_pos={0 0}, upload=1995]: Done uploading piece. err: ecclient: upload failed (node:1NyFYdUBbAhAijCDtPWB1zJCjwhgw35vEQoE4qJyg2PywRn5AH, address:77.90.31.15:4732): protocol: write tcp 159.223.143.39:45896->77.90.31.15:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:57.819353836 (2h11m13.600287229s) [seg_pos={0 0}, upload=1995]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 27 00:02:57.847511553 (2h11m13.628444923s) [upload=1995]: Done uploading segments... Jul 27 00:02:57.847557732 (2h11m13.628491099s) [upload=1995]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1m0.032577916s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [X..X.......X...X....XX..............X.X........X...XX.......] duration: 27.896911117s FAIL (2025-07-27 00:02:54 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.896935399s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...XX] duration: 30.033693002s FAIL (2025-07-27 00:02:43 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.033713717s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 30.030364293s FAIL (2025-07-27 00:02:32 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== UA3LCNXW2F7YUJ233ZW5SSU7PFQ, seg_pos={0 0}, upload=1984]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: rpc: dial tcp 104.36.82.5:4840: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:10.356708938 (2h9m26.137642303s) [seg_pos={0 0}, upload=1984]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 27 00:01:10.356794519 (2h9m26.137727884s) [upload=1984]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 27 00:01:10.356799861 (2h9m26.137733227s) [upload=1984]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 27 00:01:10.412842511 (2h9m26.193775897s) [upload=1984]: Done uploading segments... Jul 27 00:01:10.412860639 (2h9m26.193794016s) [upload=1984]: Upload finished. err: <nil> Jul 27 00:02:02.081104274 (2h10m17.862037643s) [upload=1997]: Starting upload Jul 27 00:02:02.081251072 (2h10m17.862184436s) [upload=1997]: Done starting upload Jul 27 00:02:02.081353965 (2h10m17.862287335s) [upload=1997]: Uploading segments... Jul 27 00:02:02.081859557 (2h10m17.862792937s) [upload=1997]: Got next segment. Inline: false Jul 27 00:02:02.081960739 (2h10m17.862894117s) [upload=1997]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 27 00:02:02.081967317 (2h10m17.862900686s) [upload=1997]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 27 00:02:02.109791587 (2h10m17.890724967s) [seg_pos={0 0}, upload=1997]: Begin upload segment... Jul 27 00:02:02.109836843 (2h10m17.89077021s) [seg_pos={0 0}, upload=1997]: Done begin upload segment. Jul 27 00:02:02.109861711 (2h10m17.89079509s) [noise=true , address=77.90.31.11:4554 , piece_id=CDON6KZTEAQ5HEPYUCPGKNIUUEDAPQQUJNMXPXVTJHNWHNR3SU6Q, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.109931120 (2h10m17.890864506s) [noise=true , address=104.36.82.12:4151 , piece_id=AKTJ2XIL57XAVIOKKBQ6VUIHGFUO3Y77ZTCIRPNFHPBW35ISXD2A, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.110250765 (2h10m17.891184144s) [noise=true , address=104.36.82.8:4969 , piece_id=YQPI3JIM2P3PXQ3MDOKX6PAC3XXSXAGKZRGVVKFW6VPWDTO4ZEKA, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.110442379 (2h10m17.891375765s) [noise=true , address=104.36.82.13:4187 , piece_id=6XUZ6QLTGNLAOLJUKJEBMUGRZN4WXESMUGRK2TY2OVHS5P7JYJRA, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.110619463 (2h10m17.891552838s) [noise=true , address=77.90.31.11:4553 , piece_id=7ZCRLP2DECHLTM5BTCZ7SFAP5YGI2MTUMSMAV5PNQD6XV5IYXQ4A, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.110872752 (2h10m17.891806134s) [noise=true , address=77.90.31.14:4663 , piece_id=FISRVFTJJ5EOMDT67CIY5TCUGJBRHQOSOSRSBAILRJ2MDW2W7FCA, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.111008596 (2h10m17.891941977s) [noise=true , address=45.91.116.9:4263 , piece_id=R23P77YGB3NWXJ55ZTZQB6IGLXZVFIQAMQHOPQ3CADP4F2IK3KFQ, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.111325039 (2h10m17.892258417s) [noise=true , address=77.90.31.15:4698 , piece_id=3GAZYPEG6IEUDXD4B6EGIPEC7V5RI2QKOSCTARNT5FUH3ECMWLNA, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.111514542 (2h10m17.892447923s) [noise=true , address=45.91.116.12:4225 , piece_id=P7UCY4S4R3C7GGXWGKTB6PMT2T474KDJ2H3W7KRZYIEUBJX72XVQ, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.111726483 (2h10m17.892659862s) [noise=true , address=45.91.116.7:4009 , piece_id=4WZJXUMT3EVUIYSL7UC5RV6X4RKIFFP47BGFLYBU3WWRIODVVURQ, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.111844591 (2h10m17.892777976s) [noise=true , address=77.90.31.12:4590 , piece_id=N6HN4TWGPT7YINQDENR47W2FORXGG4PX2RZPKVWMJRIFOTW4SLLQ, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.112257740 (2h10m17.893191145s) [noise=true , address=77.90.31.14:4662 , piece_id=ZFKUGFXUE4WMRAZKSTPFG3STZTN5TLLMZTZBPG65M4OAY2EGVO3Q, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.112458788 (2h10m17.893392172s) [noise=true , address=45.91.116.7:4010 , piece_id=6EUSJ4W32V62WD4FKNG5Y6CJVPRQJF4ZCSAAEZPAODWI4TIN6VCA, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.112716595 (2h10m17.893649979s) [noise=true , address=45.91.116.10:4083 , piece_id=KZSTVSEJAU4PMREKD4ZZFHILVKBBAJMAUS5VDWK25VCDXC7USTEQ, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.112841825 (2h10m17.893775207s) [noise=true , address=45.91.116.6:4973 , piece_id=TQVMQJWU7W2WKF5D7XESP677W2VU36MR35SPRPZDFLHC4KNSHPAQ, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.113114233 (2h10m17.894047617s) [noise=true , address=104.36.82.11:4115 , piece_id=BRHAQETHRLU6WA6OFRGYQCDTPAWBUTEVQHRDKR3J7ZOZBTI2OYSA, seg_pos={0 0}, upload=1997]: Uploading piece... Jul 27 00:02:02.225873984 (2h10m18.006807366s) [upload=1997]: Next returned nil segment Jul 27 00:02:02.225894604 (2h10m18.006827969s) [upload=1997]: Waiting for error group managing segments... Jul 27 00:02:02.228312435 (2h10m18.009245824s) [noise=true , address=45.91.116.7:4010 , piece_id=6EUSJ4W32V62WD4FKNG5Y6CJVPRQJF4ZCSAAEZPAODWI4TIN6VCA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:02.229102309 (2h10m18.010035689s) [noise=true , address=45.91.116.10:4083 , piece_id=KZSTVSEJAU4PMREKD4ZZFHILVKBBAJMAUS5VDWK25VCDXC7USTEQ, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:02.229791256 (2h10m18.010724638s) [noise=true , address=77.90.31.14:4662 , piece_id=ZFKUGFXUE4WMRAZKSTPFG3STZTN5TLLMZTZBPG65M4OAY2EGVO3Q, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:02.230406236 (2h10m18.011339609s) [noise=true , address=45.91.116.12:4225 , piece_id=P7UCY4S4R3C7GGXWGKTB6PMT2T474KDJ2H3W7KRZYIEUBJX72XVQ, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:02.231200277 (2h10m18.012133641s) [noise=true , address=77.90.31.11:4554 , piece_id=CDON6KZTEAQ5HEPYUCPGKNIUUEDAPQQUJNMXPXVTJHNWHNR3SU6Q, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:02.232286154 (2h10m18.013219539s) [noise=true , address=77.90.31.14:4663 , piece_id=FISRVFTJJ5EOMDT67CIY5TCUGJBRHQOSOSRSBAILRJ2MDW2W7FCA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:02.232986510 (2h10m18.013919889s) [noise=true , address=45.91.116.6:4973 , piece_id=TQVMQJWU7W2WKF5D7XESP677W2VU36MR35SPRPZDFLHC4KNSHPAQ, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:02.233587528 (2h10m18.014520896s) [noise=true , address=45.91.116.7:4009 , piece_id=4WZJXUMT3EVUIYSL7UC5RV6X4RKIFFP47BGFLYBU3WWRIODVVURQ, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:02.250677686 (2h10m18.031611064s) [noise=true , address=77.90.31.11:4553 , piece_id=7ZCRLP2DECHLTM5BTCZ7SFAP5YGI2MTUMSMAV5PNQD6XV5IYXQ4A, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:02.396939240 (2h10m18.177872604s) [noise=true , address=45.91.116.9:4263 , piece_id=R23P77YGB3NWXJ55ZTZQB6IGLXZVFIQAMQHOPQ3CADP4F2IK3KFQ, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:03.681602543 (2h10m19.462535911s) [noise=true , address=77.90.31.12:4590 , piece_id=N6HN4TWGPT7YINQDENR47W2FORXGG4PX2RZPKVWMJRIFOTW4SLLQ, seg_pos={0 0}, upload=1997]: Done uploading piece. err: <nil> Jul 27 00:02:32.048899053 (2h10m47.829832419s) [noise=true , address=104.36.82.12:4151 , piece_id=AKTJ2XIL57XAVIOKKBQ6VUIHGFUO3Y77ZTCIRPNFHPBW35ISXD2A, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:32.048963088 (2h10m47.829896454s) [noise=true , address=104.36.82.13:4187 , piece_id=6XUZ6QLTGNLAOLJUKJEBMUGRZN4WXESMUGRK2TY2OVHS5P7JYJRA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:32.049013589 (2h10m47.829946965s) [noise=true , address=77.90.31.15:4698 , piece_id=3GAZYPEG6IEUDXD4B6EGIPEC7V5RI2QKOSCTARNT5FUH3ECMWLNA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:12XVNxzmNrzdDjyJR5e6M394ptZY4sMPdH1suN42y5WynbVp8oj, address:77.90.31.15:4698): protocol: rpc: dial tcp 77.90.31.15:4698: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:32.049091588 (2h10m47.83002497s) [noise=true , address=104.36.82.11:4115 , piece_id=BRHAQETHRLU6WA6OFRGYQCDTPAWBUTEVQHRDKR3J7ZOZBTI2OYSA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:02:32.049171611 (2h10m47.830104991s) [noise=true , address=104.36.82.8:4969 , piece_id=YQPI3JIM2P3PXQ3MDOKX6PAC3XXSXAGKZRGVVKFW6VPWDTO4ZEKA, seg_pos={0 0}, upload=1997]: Done uploading piece. err: ecclient: upload failed (node:1HxTyEaQTErcZJWMYLsND9Xa8Nwaa5e1f4x2bs7UQzSRiW7buX, address:104.36.82.8:4969): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:02:32.049192849 (2h10m47.830126243s) [seg_pos={0 0}, upload=1997]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 27 00:02:32.076855520 (2h10m47.857788885s) [upload=1997]: Done uploading segments... Jul 27 00:02:32.076879200 (2h10m47.857812564s) [upload=1997]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.030394104s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X.......XXX...XX.....X.XXX...............X..X......X.XX....] duration: 570.926463ms FAIL (2025-07-27 00:02:14 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 570.953691ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...X] duration: 30.043460857s FAIL (2025-07-27 00:02:12 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 12) Duration: 30.043489951s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [.X..X.......X...X....XX..............X.X........X...XX......] duration: 27.729529625s FAIL (2025-07-27 00:01:59 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.729551816s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [XXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.028539159s FAIL (2025-07-27 00:01:57 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== segment. Jul 27 00:00:57.775236716 (2h9m13.556170102s) [noise=true , address=77.90.31.13:4644 , piece_id=B7USLVEMOWW67S2V5ELBPFBYFQEXFYJQWSXSZXM2RTGY5PKDF6TQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.775268684 (2h9m13.55620207s) [noise=true , address=64.49.13.26:4953 , piece_id=TILKJGDG6YP7AZARW4NO2UHD6RNJHAPZROT6SWD2VLI53WQ7FY2Q, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.775733963 (2h9m13.55666735s) [noise=true , address=104.36.82.46:4493 , piece_id=SHKP3MLGB3A5ABL3EB3RAG25ORWRVGAIVYKJY4MYG3HAWU6JL6HQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.775739558 (2h9m13.556672936s) [noise=true , address=77.90.31.15:4713 , piece_id=WLHB7FUEDT3D2ORHQBKD42R2QLEM2IBHQP2OTLXAVOMPU5ZETTQA, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.776244464 (2h9m13.55717785s) [noise=true , address=79.127.205.234:4610 , piece_id=UABIHF3TSXVKUXDIU4F6MDLY7UBKGUHW45R575QOOB5QTN2WUBXA, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.776298606 (2h9m13.557231985s) [noise=true , address=104.36.82.47:4528 , piece_id=EZ7TXN3VVD4M2CXGW2RENPLHPQBXO2MHXI45UV3GI4IS5D26VNTQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.776753875 (2h9m13.55768726s) [noise=true , address=79.127.205.246:4272 , piece_id=WWZENSKQVVETRJTNNE7CDDC2I6M7UFPUJMC3WMX6OQFVVZXULENQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.777180560 (2h9m13.55811394s) [noise=true , address=104.36.82.52:4089 , piece_id=CAW3CLMSY3Q7RLZFWWRPWR3PUXNA7VZTPBBTLR2HN5LVQUR4DCSQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.780407316 (2h9m13.561340701s) [noise=true , address=104.36.82.53:4129 , piece_id=S64HIMTAERGHA6MM6MA5FPWA576GL4EO5V2PCHF2GOJAREOJCAGA, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.780856964 (2h9m13.561790345s) [noise=true , address=104.36.82.56:4243 , piece_id=RS3BMQETGHRWURDAGH6EYJKQ3UXZRZWIVRLE3OR6NJILYVXZA2LA, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.781209672 (2h9m13.56214305s) [noise=true , address=104.36.82.55:4209 , piece_id=UGUI6HGMIIDS2NHYFSFSEDTAWK5MK2MKVFTTYWVCW3G27A6H7DPQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.781525117 (2h9m13.562458496s) [noise=true , address=77.90.31.11:4581 , piece_id=UM26XZ4SIHWDAYNYPZ7MKGJ4XRDBPSAB55SPZU7SWI7GF5HRYMFA, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.781890781 (2h9m13.56282416s) [noise=true , address=79.127.205.233:4643 , piece_id=7SUPFJWDQRRZORZTR5ZHK5IH47YI6JZNL2J3II3VJVQH2OAJS73Q, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.784119727 (2h9m13.565053105s) [noise=true , address=45.91.116.14:4206 , piece_id=R5ZWZGF4PQJBWY2BO4IWANJV27AWQYD3NS76DFMAEKGRYI6UFDBQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.785132722 (2h9m13.566066109s) [noise=true , address=79.127.219.38:4003 , piece_id=PG2TCCGX2URHK65ZGH6HGPL7DKGXF5L5TEVZBR3TKNCUJZL3TNRQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.785589584 (2h9m13.566522961s) [noise=true , address=64.49.13.138:4483 , piece_id=EA4VPG4UW22XVN3CH5QPOJ6RY2EYQV27FLORILQM26Q2WKNIYQNQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.785927870 (2h9m13.566861245s) [noise=true , address=77.90.31.16:4748 , piece_id=CZ5QM3M3RX27XVQVAOTIESGHKKSBY57HXSZCKZGAZ7KAIO3JARPQ, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.786322650 (2h9m13.567256031s) [noise=true , address=79.127.220.101:4041 , piece_id=G2WQS7UFY26I3UTCSQSYYH2DOEN4GMYQ5Y7U55ZW26YSV5KCX5DA, seg_pos={0 0}, upload=1979]: Uploading piece... Jul 27 00:00:57.859084590 (2h9m13.640017966s) [upload=1979]: Next returned nil segment Jul 27 00:00:57.859122614 (2h9m13.64005599s) [upload=1979]: Waiting for error group managing segments... Jul 27 00:00:57.863654302 (2h9m13.644587669s) [noise=true , address=45.91.116.14:4206 , piece_id=R5ZWZGF4PQJBWY2BO4IWANJV27AWQYD3NS76DFMAEKGRYI6UFDBQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:00:57.864360694 (2h9m13.645294073s) [noise=true , address=77.90.31.11:4581 , piece_id=UM26XZ4SIHWDAYNYPZ7MKGJ4XRDBPSAB55SPZU7SWI7GF5HRYMFA, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:00:57.901398234 (2h9m13.682331611s) [noise=true , address=79.127.220.101:4041 , piece_id=G2WQS7UFY26I3UTCSQSYYH2DOEN4GMYQ5Y7U55ZW26YSV5KCX5DA, seg_pos={0 0}, upload=1979]: Done uploading piece. err: ecclient: upload failed (node:123G35C5bhcQpSSGRtKGo78ZfxNj6bZatY3kCrdEu1kMyjV5Yiw, address:79.127.220.101:4041): protocol: storage node overloaded, request limit: 2000 Jul 27 00:00:57.929458781 (2h9m13.710392161s) [noise=true , address=77.90.31.13:4644 , piece_id=B7USLVEMOWW67S2V5ELBPFBYFQEXFYJQWSXSZXM2RTGY5PKDF6TQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:00:58.051130705 (2h9m13.832064077s) [noise=true , address=79.127.219.38:4003 , piece_id=PG2TCCGX2URHK65ZGH6HGPL7DKGXF5L5TEVZBR3TKNCUJZL3TNRQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:00:58.221274989 (2h9m14.002208364s) [noise=true , address=64.49.13.138:4483 , piece_id=EA4VPG4UW22XVN3CH5QPOJ6RY2EYQV27FLORILQM26Q2WKNIYQNQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 27 00:00:58.373358143 (2h9m14.154291517s) [noise=true , address=79.127.205.246:4272 , piece_id=WWZENSKQVVETRJTNNE7CDDC2I6M7UFPUJMC3WMX6OQFVVZXULENQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:00:58.381904170 (2h9m14.16283755s) [noise=true , address=79.127.205.233:4643 , piece_id=7SUPFJWDQRRZORZTR5ZHK5IH47YI6JZNL2J3II3VJVQH2OAJS73Q, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:00:59.087833210 (2h9m14.868766582s) [noise=true , address=79.127.205.234:4610 , piece_id=UABIHF3TSXVKUXDIU4F6MDLY7UBKGUHW45R575QOOB5QTN2WUBXA, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:01:06.359428530 (2h9m22.140361934s) [noise=true , address=77.90.31.16:4748 , piece_id=CZ5QM3M3RX27XVQVAOTIESGHKKSBY57HXSZCKZGAZ7KAIO3JARPQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:01:09.278432011 (2h9m25.059365374s) [noise=true , address=64.49.13.26:4953 , piece_id=TILKJGDG6YP7AZARW4NO2UHD6RNJHAPZROT6SWD2VLI53WQ7FY2Q, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:01:11.859875333 (2h9m27.640808702s) [noise=true , address=77.90.31.15:4713 , piece_id=WLHB7FUEDT3D2ORHQBKD42R2QLEM2IBHQP2OTLXAVOMPU5ZETTQA, seg_pos={0 0}, upload=1979]: Done uploading piece. err: <nil> Jul 27 00:01:57.706427957 (2h10m13.487361322s) [noise=true , address=104.36.82.55:4209 , piece_id=UGUI6HGMIIDS2NHYFSFSEDTAWK5MK2MKVFTTYWVCW3G27A6H7DPQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: ecclient: upload failed (node:12EmE2BgEZTG1LTsZ5p9ghqDSfg5ZQATt31Tqg5jZ5KQE2UjDnR, address:104.36.82.55:4209): protocol: write tcp 159.223.143.39:34528->104.36.82.55:4209: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:57.706497977 (2h10m13.487431342s) [noise=true , address=104.36.82.47:4528 , piece_id=EZ7TXN3VVD4M2CXGW2RENPLHPQBXO2MHXI45UV3GI4IS5D26VNTQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: ecclient: upload failed (node:12ArcCrcHdpNgBMfTcPdtTXb65RgnTh8N32ihHAx2RrddxnRYgT, address:104.36.82.47:4528): protocol: write tcp 159.223.143.39:40502->104.36.82.47:4528: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:57.706550200 (2h10m13.487483568s) [noise=true , address=104.36.82.56:4243 , piece_id=RS3BMQETGHRWURDAGH6EYJKQ3UXZRZWIVRLE3OR6NJILYVXZA2LA, seg_pos={0 0}, upload=1979]: Done uploading piece. err: ecclient: upload failed (node:12Fyogwa9NcRSUwxTPFTGL68Dbn1FEAqs8K48CNtTyKQEgVwY1c, address:104.36.82.56:4243): protocol: write tcp 159.223.143.39:36696->104.36.82.56:4243: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:57.706601741 (2h10m13.48753511s) [noise=true , address=104.36.82.46:4493 , piece_id=SHKP3MLGB3A5ABL3EB3RAG25ORWRVGAIVYKJY4MYG3HAWU6JL6HQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: ecclient: upload failed (node:127drPYZWm7RRXuCLtuzDgSaxxuu3k1tg6eo3KWtrQGjBaUbfRJ, address:104.36.82.46:4493): protocol: write tcp 159.223.143.39:39626->104.36.82.46:4493: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:57.706648689 (2h10m13.487582065s) [noise=true , address=104.36.82.52:4089 , piece_id=CAW3CLMSY3Q7RLZFWWRPWR3PUXNA7VZTPBBTLR2HN5LVQUR4DCSQ, seg_pos={0 0}, upload=1979]: Done uploading piece. err: ecclient: upload failed (node:12L2DTyJJ3v7KRjA424PVwQn9mpeq5Pq4ZTCerhrStgqgc3cVFk, address:104.36.82.52:4089): protocol: write tcp 159.223.143.39:45398->104.36.82.52:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:57.706693621 (2h10m13.487626988s) [noise=true , address=104.36.82.53:4129 , piece_id=S64HIMTAERGHA6MM6MA5FPWA576GL4EO5V2PCHF2GOJAREOJCAGA, seg_pos={0 0}, upload=1979]: Done uploading piece. err: ecclient: upload failed (node:1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, address:104.36.82.53:4129): protocol: write tcp 159.223.143.39:55136->104.36.82.53:4129: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:01:57.706711345 (2h10m13.487644724s) [seg_pos={0 0}, upload=1979]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 27 00:01:57.730740118 (2h10m13.511673491s) [upload=1979]: Done uploading segments... Jul 27 00:01:57.730770763 (2h10m13.511704142s) [upload=1979]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.028558313s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 58 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 1.847661446s FAIL (2025-07-27 00:01:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== piecestore: piecestore close: conn closed: op Jul 26 23:59:16.227057989 (2h7m32.007991372s) [seg_pos={0 0}, upload=1948]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:59:16.254547576 (2h7m32.035480941s) [upload=1948]: Done uploading segments... Jul 26 23:59:16.254573635 (2h7m32.035507007s) [upload=1948]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 23:59:54.547988211 (2h8m10.328921586s) [upload=1964]: Starting upload Jul 26 23:59:54.548042979 (2h8m10.32897636s) [upload=1964]: Done starting upload Jul 26 23:59:54.548276943 (2h8m10.32921031s) [upload=1964]: Uploading segments... Jul 26 23:59:54.548492595 (2h8m10.32942596s) [upload=1964]: Got next segment. Inline: false Jul 26 23:59:54.548548598 (2h8m10.329481973s) [upload=1964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:59:54.548554144 (2h8m10.32948752s) [upload=1964]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:59:54.579159413 (2h8m10.360092796s) [seg_pos={0 0}, upload=1964]: Begin upload segment... Jul 26 23:59:54.579239063 (2h8m10.360172445s) [seg_pos={0 0}, upload=1964]: Done begin upload segment. Jul 26 23:59:54.579255070 (2h8m10.360188454s) [noise=true , address=104.36.82.11:4115 , piece_id=5CIG5D3AQSQDMG5RSNEPWRFD2UJLONFEGJNFXUMS4UJKQJ4GLVXQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.579272031 (2h8m10.360205417s) [noise=true , address=77.90.31.17:4771 , piece_id=AWC3P5G2V5467XSG33A23UHXOTU4XCL4Y75Q36KUTLNRQBVU6YKQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.579795267 (2h8m10.360728646s) [noise=true , address=77.90.31.14:4663 , piece_id=GQFQIAEXWHPFMQPCTBXKQ2K5ZE4IF725TQZOEFBXMTEV7LUKIWVA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.580249169 (2h8m10.361182554s) [noise=true , address=45.91.116.13:4154 , piece_id=VQBGBBFJABUI2VCGZ7EZJXPRGCBUA5Q6ZEEP5O2Z4BZ6OQYRPSHQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.580720799 (2h8m10.361654184s) [noise=true , address=45.91.116.14:4191 , piece_id=HM3F4FHJEEY5D5NVYRTJXBII4YF2NZ2FP2ZWLVA63SZUOBJQWLUA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.581340224 (2h8m10.362273617s) [noise=true , address=45.91.116.10:4082 , piece_id=CROZELKMRSYASMQOYF3M2HDMEOXLDISCVZZK7J2K7WABYY5OFUTQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.581818551 (2h8m10.362751934s) [noise=true , address=104.36.82.3:4767 , piece_id=VC27LOM7GFL33XFUSQ2NSRCZMAYZGDIAPSQFWTNHACKJMFSKH4HQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.583612154 (2h8m10.364545544s) [noise=true , address=104.36.82.5:4840 , piece_id=TNKUM5SFYZRLDCLU4ULZA4CB6XXEZ74UPMMJMZPZGJRJL32NEIIA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.584156149 (2h8m10.365089533s) [noise=true , address=45.91.116.9:4261 , piece_id=TF2SAE43EGANTLH6JR2X5AWEISU66IT4UYX2MIKNXDQECGL3Y4IQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.584606999 (2h8m10.365540385s) [noise=true , address=77.90.31.21:4879 , piece_id=5ZYTXF3RSC7LZSAAXP7O6HMNLGET26COCYZWZBDZGI4GMI3VU4WQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.585024800 (2h8m10.365958185s) [noise=true , address=77.90.31.12:4592 , piece_id=F3QEZ4L6P67CKF6L4U7VMA4QPULRB2QPPFXXU5Y5KFBZHZXFESCQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.585458096 (2h8m10.366391593s) [noise=true , address=77.90.31.11:4553 , piece_id=AQWGB5IOFVUL3G3OJTMMN2B4VZDOHENPVAK5GWUY2UAYHNQ4W7XQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.585874293 (2h8m10.366807678s) [noise=true , address=104.36.82.16:4293 , piece_id=3KM2QA6ZAVLUQPYRM3OBDE7OPQFRZKW3SKIM54FNVYDOHKVRRSJA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.586299408 (2h8m10.367232794s) [noise=true , address=104.36.82.7:4912 , piece_id=6X5FNOTLHKRMV7G6CGNQIPC7P6SD5MIAEZNNZUBVFDZGUJXJUFBA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.586411303 (2h8m10.367344684s) [noise=true , address=45.91.116.5:4298 , piece_id=DBACE4YFZQVWEYCC22BXPJREKRNXO2YGWDKFTYGGI37YAOY5FZWA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.586790559 (2h8m10.367723952s) [noise=true , address=45.91.116.6:4973 , piece_id=JVBS7ADMALNAUCCN2X6WVTRJLT7USBADEQYD3EQF3VDGJMMKPMFA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.659938996 (2h8m10.440872361s) [upload=1964]: Next returned nil segment Jul 26 23:59:54.659966544 (2h8m10.44089992s) [upload=1964]: Waiting for error group managing segments... Jul 26 23:59:54.663545395 (2h8m10.444478762s) [noise=true , address=45.91.116.13:4154 , piece_id=VQBGBBFJABUI2VCGZ7EZJXPRGCBUA5Q6ZEEP5O2Z4BZ6OQYRPSHQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.663865280 (2h8m10.444798649s) [noise=true , address=45.91.116.14:4191 , piece_id=HM3F4FHJEEY5D5NVYRTJXBII4YF2NZ2FP2ZWLVA63SZUOBJQWLUA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.664358590 (2h8m10.445291955s) [noise=true , address=45.91.116.5:4298 , piece_id=DBACE4YFZQVWEYCC22BXPJREKRNXO2YGWDKFTYGGI37YAOY5FZWA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.664460777 (2h8m10.445394142s) [noise=true , address=45.91.116.6:4973 , piece_id=JVBS7ADMALNAUCCN2X6WVTRJLT7USBADEQYD3EQF3VDGJMMKPMFA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.664835941 (2h8m10.445769306s) [noise=true , address=77.90.31.11:4553 , piece_id=AQWGB5IOFVUL3G3OJTMMN2B4VZDOHENPVAK5GWUY2UAYHNQ4W7XQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.665276979 (2h8m10.446210344s) [noise=true , address=77.90.31.14:4663 , piece_id=GQFQIAEXWHPFMQPCTBXKQ2K5ZE4IF725TQZOEFBXMTEV7LUKIWVA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.665301700 (2h8m10.446235061s) [noise=true , address=45.91.116.10:4082 , piece_id=CROZELKMRSYASMQOYF3M2HDMEOXLDISCVZZK7J2K7WABYY5OFUTQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.665760740 (2h8m10.446694105s) [noise=true , address=77.90.31.12:4592 , piece_id=F3QEZ4L6P67CKF6L4U7VMA4QPULRB2QPPFXXU5Y5KFBZHZXFESCQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.665849500 (2h8m10.446782862s) [noise=true , address=45.91.116.9:4261 , piece_id=TF2SAE43EGANTLH6JR2X5AWEISU66IT4UYX2MIKNXDQECGL3Y4IQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 27 00:00:14.358891658 (2h8m30.13982504s) [noise=true , address=77.90.31.21:4879 , piece_id=5ZYTXF3RSC7LZSAAXP7O6HMNLGET26COCYZWZBDZGI4GMI3VU4WQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 27 00:00:24.507596093 (2h8m40.288529472s) [noise=true , address=104.36.82.11:4115 , piece_id=5CIG5D3AQSQDMG5RSNEPWRFD2UJLONFEGJNFXUMS4UJKQJ4GLVXQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:60098->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507722906 (2h8m40.288656283s) [noise=true , address=104.36.82.16:4293 , piece_id=3KM2QA6ZAVLUQPYRM3OBDE7OPQFRZKW3SKIM54FNVYDOHKVRRSJA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 159.223.143.39:58906->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507782037 (2h8m40.288715423s) [noise=true , address=77.90.31.17:4771 , piece_id=AWC3P5G2V5467XSG33A23UHXOTU4XCL4Y75Q36KUTLNRQBVU6YKQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:46428->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507833692 (2h8m40.288767069s) [noise=true , address=104.36.82.3:4767 , piece_id=VC27LOM7GFL33XFUSQ2NSRCZMAYZGDIAPSQFWTNHACKJMFSKH4HQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:46586->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507864769 (2h8m40.288798149s) [noise=true , address=104.36.82.7:4912 , piece_id=6X5FNOTLHKRMV7G6CGNQIPC7P6SD5MIAEZNNZUBVFDZGUJXJUFBA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:48846->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507941203 (2h8m40.288874584s) [noise=true , address=104.36.82.5:4840 , piece_id=TNKUM5SFYZRLDCLU4ULZA4CB6XXEZ74UPMMJMZPZGJRJL32NEIIA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:55006->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507976646 (2h8m40.288910036s) [seg_pos={0 0}, upload=1964]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:00:24.544497933 (2h8m40.325431318s) [upload=1964]: Done uploading segments... Jul 27 00:00:24.544530005 (2h8m40.32546339s) [upload=1964]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 1.847685926s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 55 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX...] duration: 2.016496498s FAIL (2025-07-27 00:01:10 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.016613027s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [X.X.......XXX...XX.....X.XXX...............X..X......X.XX...] duration: 1.875026191s FAIL (2025-07-27 00:01:01 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.875134387s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [XXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.026225422s FAIL (2025-07-27 00:00:57 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== =1965]: Done begin upload segment. Jul 26 23:59:57.682728020 (2h8m13.463661405s) [noise=true , address=64.49.13.154:4927 , piece_id=XIRNQBWXUA3O5NBD374HAFGZPAYOCAE2HUF3MXDF5V5BP4LYD4RA, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.682749032 (2h8m13.463682422s) [noise=true , address=79.127.205.245:4243 , piece_id=SVH2MBRURZT53RH45EN7EPMXNA3UCXPHJVNAI4QY5HPKL2ASWSQQ, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.683326904 (2h8m13.464260301s) [noise=true , address=79.127.205.235:4687 , piece_id=5H2ELRDXBVO4HQKOBDXC5MSITWDJKJM4GMTD57FYXFJRA2CKT3QQ, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.683814966 (2h8m13.46474835s) [noise=true , address=104.36.82.21:4500 , piece_id=PRXGFOHEAEOU7YAPBNKOF7M74ARS6REKXHJ6S5AMQSTORSCX747Q, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.684351431 (2h8m13.465284819s) [noise=true , address=77.90.31.18:4831 , piece_id=SPD3AOEASNV4XZSPHACV53645TNO7W2QOUELUACCTMGAO5LM52PA, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.684460147 (2h8m13.465393532s) [noise=true , address=77.90.31.13:4656 , piece_id=3UB6YMY2DE7BBWK636GNIYG5DKHETZJAP26D5UWLT7MYF6GZXKLQ, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.684801709 (2h8m13.465735102s) [noise=true , address=104.36.82.13:4198 , piece_id=GKO4JN75T4TUE3C6RX6564Z6NQILSNBPBDX7TTT6YW75KRHNPJEA, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.684949832 (2h8m13.465883209s) [noise=true , address=104.36.82.55:4211 , piece_id=AM5IN34NB3UFNL3CBOSQWQBRA3XRSTKMAOL2FHCDWBPSIRVC3O6Q, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.685187152 (2h8m13.466120534s) [noise=true , address=104.36.82.47:4535 , piece_id=EN6YEX6VNYNYBWYTFYU7NHWQ2XC4F75PPVEXR5URIB7B5YT6RW5A, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.685568763 (2h8m13.46650215s) [noise=true , address=45.91.116.10:4093 , piece_id=5CVTV47Q4KCM6ZE5CEQFBZJFQZBINBNKMKWJZWWM6744PRKKMYPQ, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.685993178 (2h8m13.466926562s) [noise=true , address=64.49.13.138:4483 , piece_id=22VCYZTMBQGJ2MO4SRJJ6IIUEGKC73VVXDORASUF7CCAHSKF4LFA, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.686099735 (2h8m13.46703312s) [noise=true , address=79.127.205.234:4589 , piece_id=5SVRERNFFOA3MSE7BNMNNDOXCQRAWEMTQLTUVEBLGXRYZESVOWOQ, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.686400363 (2h8m13.467333749s) [noise=true , address=104.36.82.54:4175 , piece_id=N2STLZTEMBON7WTGVBDRYVXP6TXIGJRLJBRRT5YOA6RKNQKNOOKA, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.686802147 (2h8m13.467735533s) [noise=true , address=104.36.82.56:4239 , piece_id=G26PSKPF7PCVDRDPIBCVYGGYEADUCCVUXZBEMBQDVVOFXFT2KYYA, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.686848171 (2h8m13.467781535s) [noise=true , address=79.127.205.246:4277 , piece_id=IJBJWKU4U75FLYD2RS7KHRWXU4LDKPZFAD25V4H7TGU4JJFZJ4ZQ, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.687242696 (2h8m13.468176095s) [noise=true , address=104.36.82.46:4514 , piece_id=7Q546ZFZD6KIMJ3FZWPO2NHPJJFSWDKXEMDOTIHQKO2AZ5AJP4RA, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.687655641 (2h8m13.468589028s) [noise=true , address=104.36.82.53:4124 , piece_id=OGO5CN52D4AO7I3URHXZLU63KJJUOIGK7XZRF2FRYOCVJVOCFHXQ, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.688053538 (2h8m13.468986956s) [noise=true , address=77.90.31.11:4587 , piece_id=XGF4KLYLGSTI5PCUL56BHSYGJD2UZA2UHWTDJ7YXKDOICIIAPYHA, seg_pos={0 0}, upload=1965]: Uploading piece... Jul 26 23:59:57.760433130 (2h8m13.541366511s) [upload=1965]: Next returned nil segment Jul 26 23:59:57.760457234 (2h8m13.541390622s) [upload=1965]: Waiting for error group managing segments... Jul 26 23:59:57.768399847 (2h8m13.549333211s) [noise=true , address=45.91.116.10:4093 , piece_id=5CVTV47Q4KCM6ZE5CEQFBZJFQZBINBNKMKWJZWWM6744PRKKMYPQ, seg_pos={0 0}, upload=1965]: Done uploading piece. err: <nil> Jul 26 23:59:57.797594051 (2h8m13.578527418s) [noise=true , address=77.90.31.13:4656 , piece_id=3UB6YMY2DE7BBWK636GNIYG5DKHETZJAP26D5UWLT7MYF6GZXKLQ, seg_pos={0 0}, upload=1965]: Done uploading piece. err: <nil> Jul 26 23:59:57.798319183 (2h8m13.579252577s) [noise=true , address=77.90.31.11:4587 , piece_id=XGF4KLYLGSTI5PCUL56BHSYGJD2UZA2UHWTDJ7YXKDOICIIAPYHA, seg_pos={0 0}, upload=1965]: Done uploading piece. err: <nil> Jul 26 23:59:57.806076236 (2h8m13.58700961s) [noise=true , address=77.90.31.18:4831 , piece_id=SPD3AOEASNV4XZSPHACV53645TNO7W2QOUELUACCTMGAO5LM52PA, seg_pos={0 0}, upload=1965]: Done uploading piece. err: <nil> Jul 26 23:59:58.258503049 (2h8m14.039436431s) [noise=true , address=79.127.205.246:4277 , piece_id=IJBJWKU4U75FLYD2RS7KHRWXU4LDKPZFAD25V4H7TGU4JJFZJ4ZQ, seg_pos={0 0}, upload=1965]: Done uploading piece. err: <nil> Jul 26 23:59:58.271072750 (2h8m14.05200612s) [noise=true , address=79.127.205.245:4243 , piece_id=SVH2MBRURZT53RH45EN7EPMXNA3UCXPHJVNAI4QY5HPKL2ASWSQQ, seg_pos={0 0}, upload=1965]: Done uploading piece. err: <nil> Jul 26 23:59:58.281779600 (2h8m14.062712975s) [noise=true , address=79.127.205.234:4589 , piece_id=5SVRERNFFOA3MSE7BNMNNDOXCQRAWEMTQLTUVEBLGXRYZESVOWOQ, seg_pos={0 0}, upload=1965]: Done uploading piece. err: <nil> Jul 26 23:59:59.596826170 (2h8m15.377759536s) [noise=true , address=79.127.205.235:4687 , piece_id=5H2ELRDXBVO4HQKOBDXC5MSITWDJKJM4GMTD57FYXFJRA2CKT3QQ, seg_pos={0 0}, upload=1965]: Done uploading piece. err: <nil> Jul 27 00:00:00.657075152 (2h8m16.438008536s) [noise=true , address=64.49.13.138:4483 , piece_id=22VCYZTMBQGJ2MO4SRJJ6IIUEGKC73VVXDORASUF7CCAHSKF4LFA, seg_pos={0 0}, upload=1965]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 27 00:00:03.005366902 (2h8m18.786300285s) [noise=true , address=64.49.13.154:4927 , piece_id=XIRNQBWXUA3O5NBD374HAFGZPAYOCAE2HUF3MXDF5V5BP4LYD4RA, seg_pos={0 0}, upload=1965]: Done uploading piece. err: <nil> Jul 27 00:00:57.610259036 (2h9m13.391192402s) [noise=true , address=104.36.82.46:4514 , piece_id=7Q546ZFZD6KIMJ3FZWPO2NHPJJFSWDKXEMDOTIHQKO2AZ5AJP4RA, seg_pos={0 0}, upload=1965]: Done uploading piece. err: ecclient: upload failed (node:1iJgoxnaKnevARfWPAkd6ygyKXHZeVH8BvJPH13zbev6mGw2Bk, address:104.36.82.46:4514): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:00:57.610931458 (2h9m13.391864824s) [noise=true , address=104.36.82.21:4500 , piece_id=PRXGFOHEAEOU7YAPBNKOF7M74ARS6REKXHJ6S5AMQSTORSCX747Q, seg_pos={0 0}, upload=1965]: Done uploading piece. err: ecclient: upload failed (node:1FPAiRym6MSHqcrk5PMK8sCafNt3sv55TrS848QJmBaUpzDcsx, address:104.36.82.21:4500): protocol: write tcp 159.223.143.39:41580->104.36.82.21:4500: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:57.610987271 (2h9m13.391920637s) [noise=true , address=104.36.82.53:4124 , piece_id=OGO5CN52D4AO7I3URHXZLU63KJJUOIGK7XZRF2FRYOCVJVOCFHXQ, seg_pos={0 0}, upload=1965]: Done uploading piece. err: ecclient: upload failed (node:12kz79BpzkFLTxbgi2Tjvz4dStT7MtrQ4GmX6bFnYw8v45jBaSy, address:104.36.82.53:4124): protocol: write tcp 159.223.143.39:46584->104.36.82.53:4124: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:57.611065051 (2h9m13.391998418s) [noise=true , address=104.36.82.47:4535 , piece_id=EN6YEX6VNYNYBWYTFYU7NHWQ2XC4F75PPVEXR5URIB7B5YT6RW5A, seg_pos={0 0}, upload=1965]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: write tcp 159.223.143.39:44246->104.36.82.47:4535: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:57.611115126 (2h9m13.392048491s) [noise=true , address=104.36.82.55:4211 , piece_id=AM5IN34NB3UFNL3CBOSQWQBRA3XRSTKMAOL2FHCDWBPSIRVC3O6Q, seg_pos={0 0}, upload=1965]: Done uploading piece. err: ecclient: upload failed (node:1269tDiDZTJSwmka4HLKMEaHErdCaUqXEYTFimGWhtjai9o4dAB, address:104.36.82.55:4211): protocol: write tcp 159.223.143.39:44870->104.36.82.55:4211: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:57.611196404 (2h9m13.392129768s) [noise=true , address=104.36.82.54:4175 , piece_id=N2STLZTEMBON7WTGVBDRYVXP6TXIGJRLJBRRT5YOA6RKNQKNOOKA, seg_pos={0 0}, upload=1965]: Done uploading piece. err: ecclient: upload failed (node:12ou7jwFyJP3mNoALddniUTDT9duVyQKZjeq4xHCyx6sRWpwEPJ, address:104.36.82.54:4175): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:00:57.611281104 (2h9m13.392214469s) [noise=true , address=104.36.82.13:4198 , piece_id=GKO4JN75T4TUE3C6RX6564Z6NQILSNBPBDX7TTT6YW75KRHNPJEA, seg_pos={0 0}, upload=1965]: Done uploading piece. err: ecclient: upload failed (node:1Fj2mNZexPwM4B48kxLin8GF17KJN1RxNcZjGkD36LDviUevFm, address:104.36.82.13:4198): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:00:57.611371624 (2h9m13.39230499s) [noise=true , address=104.36.82.56:4239 , piece_id=G26PSKPF7PCVDRDPIBCVYGGYEADUCCVUXZBEMBQDVVOFXFT2KYYA, seg_pos={0 0}, upload=1965]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: context deadline exceeded; context deadline exceeded Jul 27 00:00:57.611388831 (2h9m13.392322209s) [seg_pos={0 0}, upload=1965]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 27 00:00:57.634432783 (2h9m13.415366158s) [upload=1965]: Done uploading segments... Jul 27 00:00:57.634458669 (2h9m13.415392045s) [upload=1965]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.026243109s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [..X..X.......X...X....XX..............X.X........X...XX.....] duration: 27.693206352s FAIL (2025-07-27 00:00:54 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 27.693224058s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [.X.X.......XXX...XX.....X.XXX...............X..X......X.XX..] duration: 2.218024143s FAIL (2025-07-27 00:00:39 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.218054912s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [...X..X.......X...X....XX..............X.X........X...XX....] duration: 54.889232269s FAIL (2025-07-27 00:00:26 UTC) operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 54.889326144s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.040368781s FAIL (2025-07-27 00:00:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== piecestore: piecestore close: conn closed: op Jul 26 23:59:16.227057989 (2h7m32.007991372s) [seg_pos={0 0}, upload=1948]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:59:16.254547576 (2h7m32.035480941s) [upload=1948]: Done uploading segments... Jul 26 23:59:16.254573635 (2h7m32.035507007s) [upload=1948]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Jul 26 23:59:54.547988211 (2h8m10.328921586s) [upload=1964]: Starting upload Jul 26 23:59:54.548042979 (2h8m10.32897636s) [upload=1964]: Done starting upload Jul 26 23:59:54.548276943 (2h8m10.32921031s) [upload=1964]: Uploading segments... Jul 26 23:59:54.548492595 (2h8m10.32942596s) [upload=1964]: Got next segment. Inline: false Jul 26 23:59:54.548548598 (2h8m10.329481973s) [upload=1964]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:59:54.548554144 (2h8m10.32948752s) [upload=1964]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:59:54.579159413 (2h8m10.360092796s) [seg_pos={0 0}, upload=1964]: Begin upload segment... Jul 26 23:59:54.579239063 (2h8m10.360172445s) [seg_pos={0 0}, upload=1964]: Done begin upload segment. Jul 26 23:59:54.579255070 (2h8m10.360188454s) [noise=true , address=104.36.82.11:4115 , piece_id=5CIG5D3AQSQDMG5RSNEPWRFD2UJLONFEGJNFXUMS4UJKQJ4GLVXQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.579272031 (2h8m10.360205417s) [noise=true , address=77.90.31.17:4771 , piece_id=AWC3P5G2V5467XSG33A23UHXOTU4XCL4Y75Q36KUTLNRQBVU6YKQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.579795267 (2h8m10.360728646s) [noise=true , address=77.90.31.14:4663 , piece_id=GQFQIAEXWHPFMQPCTBXKQ2K5ZE4IF725TQZOEFBXMTEV7LUKIWVA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.580249169 (2h8m10.361182554s) [noise=true , address=45.91.116.13:4154 , piece_id=VQBGBBFJABUI2VCGZ7EZJXPRGCBUA5Q6ZEEP5O2Z4BZ6OQYRPSHQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.580720799 (2h8m10.361654184s) [noise=true , address=45.91.116.14:4191 , piece_id=HM3F4FHJEEY5D5NVYRTJXBII4YF2NZ2FP2ZWLVA63SZUOBJQWLUA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.581340224 (2h8m10.362273617s) [noise=true , address=45.91.116.10:4082 , piece_id=CROZELKMRSYASMQOYF3M2HDMEOXLDISCVZZK7J2K7WABYY5OFUTQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.581818551 (2h8m10.362751934s) [noise=true , address=104.36.82.3:4767 , piece_id=VC27LOM7GFL33XFUSQ2NSRCZMAYZGDIAPSQFWTNHACKJMFSKH4HQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.583612154 (2h8m10.364545544s) [noise=true , address=104.36.82.5:4840 , piece_id=TNKUM5SFYZRLDCLU4ULZA4CB6XXEZ74UPMMJMZPZGJRJL32NEIIA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.584156149 (2h8m10.365089533s) [noise=true , address=45.91.116.9:4261 , piece_id=TF2SAE43EGANTLH6JR2X5AWEISU66IT4UYX2MIKNXDQECGL3Y4IQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.584606999 (2h8m10.365540385s) [noise=true , address=77.90.31.21:4879 , piece_id=5ZYTXF3RSC7LZSAAXP7O6HMNLGET26COCYZWZBDZGI4GMI3VU4WQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.585024800 (2h8m10.365958185s) [noise=true , address=77.90.31.12:4592 , piece_id=F3QEZ4L6P67CKF6L4U7VMA4QPULRB2QPPFXXU5Y5KFBZHZXFESCQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.585458096 (2h8m10.366391593s) [noise=true , address=77.90.31.11:4553 , piece_id=AQWGB5IOFVUL3G3OJTMMN2B4VZDOHENPVAK5GWUY2UAYHNQ4W7XQ, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.585874293 (2h8m10.366807678s) [noise=true , address=104.36.82.16:4293 , piece_id=3KM2QA6ZAVLUQPYRM3OBDE7OPQFRZKW3SKIM54FNVYDOHKVRRSJA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.586299408 (2h8m10.367232794s) [noise=true , address=104.36.82.7:4912 , piece_id=6X5FNOTLHKRMV7G6CGNQIPC7P6SD5MIAEZNNZUBVFDZGUJXJUFBA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.586411303 (2h8m10.367344684s) [noise=true , address=45.91.116.5:4298 , piece_id=DBACE4YFZQVWEYCC22BXPJREKRNXO2YGWDKFTYGGI37YAOY5FZWA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.586790559 (2h8m10.367723952s) [noise=true , address=45.91.116.6:4973 , piece_id=JVBS7ADMALNAUCCN2X6WVTRJLT7USBADEQYD3EQF3VDGJMMKPMFA, seg_pos={0 0}, upload=1964]: Uploading piece... Jul 26 23:59:54.659938996 (2h8m10.440872361s) [upload=1964]: Next returned nil segment Jul 26 23:59:54.659966544 (2h8m10.44089992s) [upload=1964]: Waiting for error group managing segments... Jul 26 23:59:54.663545395 (2h8m10.444478762s) [noise=true , address=45.91.116.13:4154 , piece_id=VQBGBBFJABUI2VCGZ7EZJXPRGCBUA5Q6ZEEP5O2Z4BZ6OQYRPSHQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.663865280 (2h8m10.444798649s) [noise=true , address=45.91.116.14:4191 , piece_id=HM3F4FHJEEY5D5NVYRTJXBII4YF2NZ2FP2ZWLVA63SZUOBJQWLUA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.664358590 (2h8m10.445291955s) [noise=true , address=45.91.116.5:4298 , piece_id=DBACE4YFZQVWEYCC22BXPJREKRNXO2YGWDKFTYGGI37YAOY5FZWA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.664460777 (2h8m10.445394142s) [noise=true , address=45.91.116.6:4973 , piece_id=JVBS7ADMALNAUCCN2X6WVTRJLT7USBADEQYD3EQF3VDGJMMKPMFA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.664835941 (2h8m10.445769306s) [noise=true , address=77.90.31.11:4553 , piece_id=AQWGB5IOFVUL3G3OJTMMN2B4VZDOHENPVAK5GWUY2UAYHNQ4W7XQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.665276979 (2h8m10.446210344s) [noise=true , address=77.90.31.14:4663 , piece_id=GQFQIAEXWHPFMQPCTBXKQ2K5ZE4IF725TQZOEFBXMTEV7LUKIWVA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.665301700 (2h8m10.446235061s) [noise=true , address=45.91.116.10:4082 , piece_id=CROZELKMRSYASMQOYF3M2HDMEOXLDISCVZZK7J2K7WABYY5OFUTQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.665760740 (2h8m10.446694105s) [noise=true , address=77.90.31.12:4592 , piece_id=F3QEZ4L6P67CKF6L4U7VMA4QPULRB2QPPFXXU5Y5KFBZHZXFESCQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 26 23:59:54.665849500 (2h8m10.446782862s) [noise=true , address=45.91.116.9:4261 , piece_id=TF2SAE43EGANTLH6JR2X5AWEISU66IT4UYX2MIKNXDQECGL3Y4IQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 27 00:00:14.358891658 (2h8m30.13982504s) [noise=true , address=77.90.31.21:4879 , piece_id=5ZYTXF3RSC7LZSAAXP7O6HMNLGET26COCYZWZBDZGI4GMI3VU4WQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: <nil> Jul 27 00:00:24.507596093 (2h8m40.288529472s) [noise=true , address=104.36.82.11:4115 , piece_id=5CIG5D3AQSQDMG5RSNEPWRFD2UJLONFEGJNFXUMS4UJKQJ4GLVXQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:60098->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507722906 (2h8m40.288656283s) [noise=true , address=104.36.82.16:4293 , piece_id=3KM2QA6ZAVLUQPYRM3OBDE7OPQFRZKW3SKIM54FNVYDOHKVRRSJA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: write tcp 159.223.143.39:58906->104.36.82.16:4293: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507782037 (2h8m40.288715423s) [noise=true , address=77.90.31.17:4771 , piece_id=AWC3P5G2V5467XSG33A23UHXOTU4XCL4Y75Q36KUTLNRQBVU6YKQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:12AxqL8pD57cAVRsoNMBsqjWCvRwnKYwb9ESatQQZLRxFEGCjq2, address:77.90.31.17:4771): protocol: write tcp 159.223.143.39:46428->77.90.31.17:4771: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507833692 (2h8m40.288767069s) [noise=true , address=104.36.82.3:4767 , piece_id=VC27LOM7GFL33XFUSQ2NSRCZMAYZGDIAPSQFWTNHACKJMFSKH4HQ, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: write tcp 159.223.143.39:46586->104.36.82.3:4767: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507864769 (2h8m40.288798149s) [noise=true , address=104.36.82.7:4912 , piece_id=6X5FNOTLHKRMV7G6CGNQIPC7P6SD5MIAEZNNZUBVFDZGUJXJUFBA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:48846->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507941203 (2h8m40.288874584s) [noise=true , address=104.36.82.5:4840 , piece_id=TNKUM5SFYZRLDCLU4ULZA4CB6XXEZ74UPMMJMZPZGJRJL32NEIIA, seg_pos={0 0}, upload=1964]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:55006->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 27 00:00:24.507976646 (2h8m40.288910036s) [seg_pos={0 0}, upload=1964]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 27 00:00:24.544497933 (2h8m40.325431318s) [upload=1964]: Done uploading segments... Jul 27 00:00:24.544530005 (2h8m40.32546339s) [upload=1964]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.0403965s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 56 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXX.XXXXXXXX..] duration: 2.950845546s FAIL (2025-07-27 00:00:22 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 2.950880206s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 58 out of 60 is failed (threshold=3) [XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.027596837s FAIL (2025-07-26 23:59:57 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== [upload=1950]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:58:57.583123083 (2h7m13.364056459s) [seg_pos={0 0}, upload=1950]: Begin upload segment... Jul 26 23:58:57.583170346 (2h7m13.364103712s) [seg_pos={0 0}, upload=1950]: Done begin upload segment. Jul 26 23:58:57.583216747 (2h7m13.364150129s) [noise=true , address=79.127.220.101:4041 , piece_id=F2TOFKHQMXQXXWTD7NRQWURXY4HTT77SYPISQLOQNXEHBMKK43GQ, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.583228177 (2h7m13.364161544s) [noise=true , address=79.127.220.97:4361 , piece_id=LUUTJS4JTBIMZJEK7RSJSZGMJMVKPYRKFQP34FJNM4M45MKFDC5Q, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.583592965 (2h7m13.364526343s) [noise=true , address=45.91.116.10:4093 , piece_id=JUIABMNKE2L5PRAV6TIFMOAP4R7XVDWHWJ2RLI77KIQJXUDJFVWQ, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.583700975 (2h7m13.364634341s) [noise=true , address=64.49.13.138:4482 , piece_id=FKOQE2FWUYFHDY7ODSQKP5AVWPY5EFMBCKJHRDJ3IQPDVWZIWGBA, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.583873420 (2h7m13.364806798s) [noise=true , address=156.146.43.225:4434 , piece_id=COMVSQHZJHENA7VL6N67EY4URHYTCTJ3JQ4SAD2K7WJVWMZSJQMQ, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.584294208 (2h7m13.365227587s) [noise=true , address=104.36.82.54:4171 , piece_id=B75MPJG2NPUENK744IXC6S5AXVTNASUVNCJN3WSY23I6IYGFLMJQ, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.584525691 (2h7m13.365459077s) [noise=true , address=104.36.82.41:4771 , piece_id=2DIYM6YBACQFHHIL6SW3JFIVTBGEUQXLXIEYEJLBOXZTXJPB66EQ, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.584627343 (2h7m13.36556079s) [noise=true , address=77.90.31.11:4577 , piece_id=ZR6A4LCNVRCU7SPCD3MOMCX5TAPEEL6DBPJZ5J4F2DPN57AYIL2Q, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.584922118 (2h7m13.365855493s) [noise=true , address=45.91.116.14:4206 , piece_id=JTXNBWRFLANQQFVUPI66BFFRQXU3IOE2YGAPJY7UR473VRFH3PBA, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.585187624 (2h7m13.366121002s) [noise=true , address=104.36.82.22:4514 , piece_id=CJHIGGFRC7HSQ2Y36B3YDIK5VEBMDEVUGHUZHK5SMI4HXLZWSC2Q, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.585791738 (2h7m13.366725119s) [noise=true , address=79.127.219.38:4018 , piece_id=AKS3NWDKR627J7G2NPDL3SKVBJGKQSOZORRWE6MXFF4V67UTUNJA, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.587457874 (2h7m13.368391255s) [noise=true , address=77.90.31.13:4643 , piece_id=ZYEVYXPWWWYTA7EWXDMACWMNOGZSU7J2XM7CFQV2MMWS3XBMIO5A, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.587824571 (2h7m13.368757946s) [noise=true , address=77.90.31.19:4847 , piece_id=ZTBX2KUWLDR4WUXYBAYBOUGDCSXZU5ZLEAXJPODYCZFTPM4AJVLA, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.588296379 (2h7m13.36922976s) [noise=true , address=77.90.31.16:4743 , piece_id=NJQTG5ULV2KGDFKWPEPAIU26REODAL4SQJBSUYCXKBWECQHP2GVA, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.588691729 (2h7m13.369625114s) [noise=true , address=104.36.82.47:4535 , piece_id=JW5CLQGCPYMQI5SDOFT7O76TLJEYY5J63EIWEELM2CM7BXLZOOFQ, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.589038441 (2h7m13.369971817s) [noise=true , address=77.90.31.10:4540 , piece_id=P6LZTNLNMUGVK3YQYS7EUDAAPLRDSFRS3XT6ASBDD4LHH437BZCA, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.589360434 (2h7m13.370293809s) [noise=true , address=104.36.82.53:4127 , piece_id=UBIYHMZR5OCUEFUEISHZWY2QLPFQEBLIAEDO2HJ42RF7WFQN52YA, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.589664794 (2h7m13.370598163s) [noise=true , address=104.36.82.46:4515 , piece_id=627GFTI4FBA6ETH7ABQ7GWF77LQMVIITEDLAOAIIB4BX2RQT5BEQ, seg_pos={0 0}, upload=1950]: Uploading piece... Jul 26 23:58:57.644877694 (2h7m13.425811063s) [upload=1950]: Next returned nil segment Jul 26 23:58:57.644893910 (2h7m13.425827275s) [upload=1950]: Waiting for error group managing segments... Jul 26 23:58:57.648243130 (2h7m13.429176499s) [noise=true , address=79.127.219.38:4018 , piece_id=AKS3NWDKR627J7G2NPDL3SKVBJGKQSOZORRWE6MXFF4V67UTUNJA, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:58:57.686307174 (2h7m13.467240544s) [noise=true , address=77.90.31.13:4643 , piece_id=ZYEVYXPWWWYTA7EWXDMACWMNOGZSU7J2XM7CFQV2MMWS3XBMIO5A, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:58:57.820828523 (2h7m13.60176189s) [noise=true , address=77.90.31.11:4577 , piece_id=ZR6A4LCNVRCU7SPCD3MOMCX5TAPEEL6DBPJZ5J4F2DPN57AYIL2Q, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:58:58.061614895 (2h7m13.842548275s) [noise=true , address=156.146.43.225:4434 , piece_id=COMVSQHZJHENA7VL6N67EY4URHYTCTJ3JQ4SAD2K7WJVWMZSJQMQ, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:58:58.353070332 (2h7m14.134003711s) [noise=true , address=77.90.31.16:4743 , piece_id=NJQTG5ULV2KGDFKWPEPAIU26REODAL4SQJBSUYCXKBWECQHP2GVA, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:58:58.597956357 (2h7m14.378889728s) [noise=true , address=45.91.116.14:4206 , piece_id=JTXNBWRFLANQQFVUPI66BFFRQXU3IOE2YGAPJY7UR473VRFH3PBA, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:58:58.870535624 (2h7m14.651468988s) [noise=true , address=77.90.31.19:4847 , piece_id=ZTBX2KUWLDR4WUXYBAYBOUGDCSXZU5ZLEAXJPODYCZFTPM4AJVLA, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:58:59.468270730 (2h7m15.249204099s) [noise=true , address=45.91.116.10:4093 , piece_id=JUIABMNKE2L5PRAV6TIFMOAP4R7XVDWHWJ2RLI77KIQJXUDJFVWQ, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:59:00.597144452 (2h7m16.378077826s) [noise=true , address=79.127.220.101:4041 , piece_id=F2TOFKHQMXQXXWTD7NRQWURXY4HTT77SYPISQLOQNXEHBMKK43GQ, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:59:00.810801926 (2h7m16.591735293s) [noise=true , address=79.127.220.97:4361 , piece_id=LUUTJS4JTBIMZJEK7RSJSZGMJMVKPYRKFQP34FJNM4M45MKFDC5Q, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:59:03.813847724 (2h7m19.594781091s) [noise=true , address=64.49.13.138:4482 , piece_id=FKOQE2FWUYFHDY7ODSQKP5AVWPY5EFMBCKJHRDJ3IQPDVWZIWGBA, seg_pos={0 0}, upload=1950]: Done uploading piece. err: <nil> Jul 26 23:59:57.515591091 (2h8m13.296524466s) [noise=true , address=104.36.82.54:4171 , piece_id=B75MPJG2NPUENK744IXC6S5AXVTNASUVNCJN3WSY23I6IYGFLMJQ, seg_pos={0 0}, upload=1950]: Done uploading piece. err: ecclient: upload failed (node:125SMhDsEZruNu6LkmQn2M7R6WzvXTdk86DRsKf522u8BNiANq3, address:104.36.82.54:4171): protocol: rpc: dial tcp 104.36.82.54:4171: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:57.515632924 (2h8m13.296566302s) [noise=true , address=104.36.82.47:4535 , piece_id=JW5CLQGCPYMQI5SDOFT7O76TLJEYY5J63EIWEELM2CM7BXLZOOFQ, seg_pos={0 0}, upload=1950]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: rpc: dial tcp 104.36.82.47:4535: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:57.515746678 (2h8m13.296680044s) [noise=true , address=104.36.82.53:4127 , piece_id=UBIYHMZR5OCUEFUEISHZWY2QLPFQEBLIAEDO2HJ42RF7WFQN52YA, seg_pos={0 0}, upload=1950]: Done uploading piece. err: ecclient: upload failed (node:12VM12t6Dq6SyApviDYSSUqWnnzW3wTfA4o8VsvJaah4fLj9EWh, address:104.36.82.53:4127): protocol: rpc: dial tcp 104.36.82.53:4127: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:57.515781388 (2h8m13.296714755s) [noise=true , address=104.36.82.41:4771 , piece_id=2DIYM6YBACQFHHIL6SW3JFIVTBGEUQXLXIEYEJLBOXZTXJPB66EQ, seg_pos={0 0}, upload=1950]: Done uploading piece. err: ecclient: upload failed (node:1PggYtrfQ5ozsJL1HeAVJWc4jLvf5jiHspX3CgFY89swMimWto, address:104.36.82.41:4771): protocol: rpc: dial tcp 104.36.82.41:4771: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:57.515825081 (2h8m13.296758459s) [noise=true , address=104.36.82.22:4514 , piece_id=CJHIGGFRC7HSQ2Y36B3YDIK5VEBMDEVUGHUZHK5SMI4HXLZWSC2Q, seg_pos={0 0}, upload=1950]: Done uploading piece. err: ecclient: upload failed (node:12Nh9nHAnKWgu1ggCAZ7WcUadbvsPqAgbqzGrBnSaDMqDhquQ4R, address:104.36.82.22:4514): protocol: rpc: dial tcp 104.36.82.22:4514: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:57.515849512 (2h8m13.296782878s) [noise=true , address=104.36.82.46:4515 , piece_id=627GFTI4FBA6ETH7ABQ7GWF77LQMVIITEDLAOAIIB4BX2RQT5BEQ, seg_pos={0 0}, upload=1950]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: write tcp 159.223.143.39:59944->104.36.82.46:4515: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:57.515909624 (2h8m13.296843002s) [noise=true , address=77.90.31.10:4540 , piece_id=P6LZTNLNMUGVK3YQYS7EUDAAPLRDSFRS3XT6ASBDD4LHH437BZCA, seg_pos={0 0}, upload=1950]: Done uploading piece. err: ecclient: upload failed (node:1UUkKs1GGD9A6byJ1ZNZuZoX4kaa6zAdAKsaKs8bZ5A2jsxreA, address:77.90.31.10:4540): protocol: read tcp 159.223.143.39:36912->77.90.31.10:4540: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:57.515945737 (2h8m13.296879121s) [seg_pos={0 0}, upload=1950]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:59:57.540351536 (2h8m13.321284914s) [upload=1950]: Done uploading segments... Jul 26 23:59:57.540379449 (2h8m13.321312825s) [upload=1950]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration:* 1m0.027629046s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [X.X.X.......XXX...XX.....X.XXX...............X..X......X.XX.] duration: 2.069330336s FAIL (2025-07-26 23:59:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.069365405s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [.X.X.X.......XXX...XX.....X.XXX...............X..X......X.XX] duration: 30.00108502s FAIL (2025-07-26 23:59:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.00111167s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 59 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.030408772s FAIL (2025-07-26 23:59:16 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:58:20.641666399 (2h6m36.422599768s) [noise=true , address=104.36.82.16:4293 , piece_id=D2F6MCHAFFAYF2TFXGS3Q6VLQPDLK63Y2B5YPDLSSFII2BV3N3PA, seg_pos={0 0}, upload=1933]: Done uploading piece. err: ecclient: upload failed (node:12jsNwKfNEjPG9fmLdvdJwMYPHeh5w3KtWA1MsXZ9Ns8QBpVYbE, address:104.36.82.16:4293): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:58:20.641682855 (2h6m36.422616295s) [seg_pos={0 0}, upload=1933]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:58:20.669064353 (2h6m36.449997721s) [upload=1933]: Done uploading segments... Jul 26 23:58:20.669089437 (2h6m36.450022813s) [upload=1933]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Jul 26 23:58:46.255943658 (2h7m2.036877023s) [upload=1948]: Starting upload Jul 26 23:58:46.256038553 (2h7m2.036971929s) [upload=1948]: Done starting upload Jul 26 23:58:46.256394637 (2h7m2.037328002s) [upload=1948]: Uploading segments... Jul 26 23:58:46.256483799 (2h7m2.037417167s) [upload=1948]: Got next segment. Inline: false Jul 26 23:58:46.256544434 (2h7m2.03747781s) [upload=1948]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:58:46.256549757 (2h7m2.037483134s) [upload=1948]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:58:46.284545335 (2h7m2.065478719s) [seg_pos={0 0}, upload=1948]: Begin upload segment... Jul 26 23:58:46.284616775 (2h7m2.065550155s) [seg_pos={0 0}, upload=1948]: Done begin upload segment. Jul 26 23:58:46.284640254 (2h7m2.065573636s) [noise=true , address=45.91.116.6:4973 , piece_id=YQVSXOAHSXNCLA76CWCPZHTWXOVCGURZOMTLPBDCLXN7RPKBHDOQ, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.284656200 (2h7m2.065589585s) [noise=true , address=77.90.31.17:4770 , piece_id=U2LN2MD3OP5ZMFGRJM2CMLN4UMMTH32QYGU2SDYJIGS3DEJELWBA, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.284729873 (2h7m2.065663241s) [noise=true , address=45.91.116.4:4938 , piece_id=GMT26RMQ5XZZQCYIKXOS22F5BWNTAWZZV3HXYOH7DBFPY6FXAR3Q, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.285201288 (2h7m2.066134671s) [noise=true , address=45.91.116.6:4974 , piece_id=TXCD6KERV6UMKSOMY76IG63BENK6BZP65JSPG5WXDQRX22QFTHAA, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.285308763 (2h7m2.066242145s) [noise=true , address=104.36.82.12:4150 , piece_id=M2VVRAEHXNG7I7TUZ3BVHEGEM6UDUURWF524XW2FNBEH5EPWGPYQ, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.285591704 (2h7m2.066525081s) [noise=true , address=45.91.116.7:4011 , piece_id=TEKFCJKCQY3THCEF6EIO53P53MUZ5YGM6OLDZR2LTB4H75GSY3CA, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.285889973 (2h7m2.06682337s) [noise=true , address=77.90.31.12:4589 , piece_id=C5HNYF4OJ6I5Y45PTVUDPDUVGSHLKLKQAF2GFOQBBJXOKH3MCLMQ, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.286436892 (2h7m2.067370268s) [noise=true , address=45.91.116.7:4009 , piece_id=TWIWHK2VZ5VK7Z43ZPHZAUELEZSDO7CLJHWHRMVTATQOR4DQZMEA, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.298540690 (2h7m2.079474078s) [noise=true , address=104.36.82.2:4732 , piece_id=WOEDYQMH5F3MTAKXT4TAXXJXCUYE6GSO5P2N25DQOP4UUKEIQR4A, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.308351248 (2h7m2.089284634s) [noise=true , address=104.36.82.11:4114 , piece_id=FJ34CYR67HTLNPOS2BNXR6VJTUCDJNSI6SV2LZWMJTWCQ7GA3HSQ, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.313269976 (2h7m2.094203405s) [noise=true , address=104.36.82.14:4223 , piece_id=54VKHSI5IM33P6LR7LJ5YI5RXS4T7OSQCDV262KT6W5AR5AC6AHA, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.318023912 (2h7m2.0989573s) [noise=true , address=77.90.31.15:4699 , piece_id=KWFPLPYY3H6ZVQBADQQJGIUH6ZNG7DMJF3KMRQTAECXYMCCZVTLQ, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.321982649 (2h7m2.102916036s) [noise=true , address=77.90.31.13:4628 , piece_id=U2FJZCNGDIMHFTOVPOB5AN33TFB4I22BCYWATPURFGMJ4XXTCUMQ, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.325058306 (2h7m2.105991685s) [noise=true , address=45.91.116.9:4262 , piece_id=KPKD2TZJOHOF4CQIORUZGNE7ZGMEI6D3LGBTC37BNXTPC7HR7M4A, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.328605517 (2h7m2.109538902s) [noise=true , address=77.90.31.14:4663 , piece_id=ZLCDRENFFQOP7PNBX3EAJQUGGBYCFMHZA3FAJVUAU2JUKMSIPZFA, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.329095101 (2h7m2.110028469s) [noise=true , address=77.90.31.18:4805 , piece_id=FLNICJ2VW2O7VQIFGPPKJYYPRN3RYC725F4LPJSEE5EVVULUTQVQ, seg_pos={0 0}, upload=1948]: Uploading piece... Jul 26 23:58:46.397566505 (2h7m2.178499877s) [upload=1948]: Next returned nil segment Jul 26 23:58:46.397582395 (2h7m2.178515763s) [upload=1948]: Waiting for error group managing segments... Jul 26 23:58:46.415801036 (2h7m2.196734406s) [noise=true , address=77.90.31.14:4663 , piece_id=ZLCDRENFFQOP7PNBX3EAJQUGGBYCFMHZA3FAJVUAU2JUKMSIPZFA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.415841301 (2h7m2.196774664s) [noise=true , address=45.91.116.6:4974 , piece_id=TXCD6KERV6UMKSOMY76IG63BENK6BZP65JSPG5WXDQRX22QFTHAA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.416417702 (2h7m2.197351069s) [noise=true , address=45.91.116.7:4011 , piece_id=TEKFCJKCQY3THCEF6EIO53P53MUZ5YGM6OLDZR2LTB4H75GSY3CA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.418202539 (2h7m2.199135922s) [noise=true , address=77.90.31.12:4589 , piece_id=C5HNYF4OJ6I5Y45PTVUDPDUVGSHLKLKQAF2GFOQBBJXOKH3MCLMQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.418683787 (2h7m2.199617156s) [noise=true , address=45.91.116.7:4009 , piece_id=TWIWHK2VZ5VK7Z43ZPHZAUELEZSDO7CLJHWHRMVTATQOR4DQZMEA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.427339346 (2h7m2.208272725s) [noise=true , address=77.90.31.15:4699 , piece_id=KWFPLPYY3H6ZVQBADQQJGIUH6ZNG7DMJF3KMRQTAECXYMCCZVTLQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.442872874 (2h7m2.22380625s) [noise=true , address=45.91.116.9:4262 , piece_id=KPKD2TZJOHOF4CQIORUZGNE7ZGMEI6D3LGBTC37BNXTPC7HR7M4A, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.578499304 (2h7m2.359432672s) [noise=true , address=45.91.116.6:4973 , piece_id=YQVSXOAHSXNCLA76CWCPZHTWXOVCGURZOMTLPBDCLXN7RPKBHDOQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.624214120 (2h7m2.405147492s) [noise=true , address=45.91.116.4:4938 , piece_id=GMT26RMQ5XZZQCYIKXOS22F5BWNTAWZZV3HXYOH7DBFPY6FXAR3Q, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.637977699 (2h7m2.418911076s) [noise=true , address=77.90.31.18:4805 , piece_id=FLNICJ2VW2O7VQIFGPPKJYYPRN3RYC725F4LPJSEE5EVVULUTQVQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:58:46.841315447 (2h7m2.622248825s) [noise=true , address=77.90.31.13:4628 , piece_id=U2FJZCNGDIMHFTOVPOB5AN33TFB4I22BCYWATPURFGMJ4XXTCUMQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: <nil> Jul 26 23:59:16.226795168 (2h7m32.007728556s) [noise=true , address=104.36.82.11:4114 , piece_id=FJ34CYR67HTLNPOS2BNXR6VJTUCDJNSI6SV2LZWMJTWCQ7GA3HSQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:16.226882604 (2h7m32.007815988s) [noise=true , address=104.36.82.14:4223 , piece_id=54VKHSI5IM33P6LR7LJ5YI5RXS4T7OSQCDV262KT6W5AR5AC6AHA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: rpc: dial tcp 104.36.82.14:4223: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:16.226921080 (2h7m32.007854442s) [noise=true , address=77.90.31.17:4770 , piece_id=U2LN2MD3OP5ZMFGRJM2CMLN4UMMTH32QYGU2SDYJIGS3DEJELWBA, seg_pos={0 0}, upload=1948]: Done uploading piece. err: ecclient: upload failed (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R, address:77.90.31.17:4770): protocol: rpc: dial tcp 77.90.31.17:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:16.226973654 (2h7m32.007907037s) [noise=true , address=104.36.82.2:4732 , piece_id=WOEDYQMH5F3MTAKXT4TAXXJXCUYE6GSO5P2N25DQOP4UUKEIQR4A, seg_pos={0 0}, upload=1948]: Done uploading piece. err: ecclient: upload failed (node:12KuoS4Fh3YKpoEgZuxcDLE1e1KZENgrLTqR9T5yCWonHiVfaXE, address:104.36.82.2:4732): protocol: write tcp 159.223.143.39:36558->104.36.82.2:4732: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:16.226988830 (2h7m32.007922215s) [noise=true , address=104.36.82.12:4150 , piece_id=M2VVRAEHXNG7I7TUZ3BVHEGEM6UDUURWF524XW2FNBEH5EPWGPYQ, seg_pos={0 0}, upload=1948]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:47142->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:59:16.227057989 (2h7m32.007991372s) [seg_pos={0 0}, upload=1948]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:59:16.254547576 (2h7m32.035480941s) [upload=1948]: Done uploading segments... Jul 26 23:59:16.254573635 (2h7m32.035507007s) [upload=1948]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.030434838s core uplink us-select-1

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)