Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [......X.X....X.XX.....................X.....................] duration: 516.564639ms FAIL (2025-07-28 01:54:12 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 516.602165ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 29 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX...............................] duration: 1.353943628s FAIL (2025-07-28 01:53:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1.353978252s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [................X..XX.X..X.X................................] duration: 514.422965ms FAIL (2025-07-28 01:53:30 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 514.461026ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......X.X....X.XX.....................X....................] duration: 496.960545ms FAIL (2025-07-28 01:53:20 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 497.000582ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 30 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..............................] duration: 1.062582702s FAIL (2025-07-28 01:53:18 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1.06261383s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.................X..XX.X..X.X...............................] duration: 493.006674ms FAIL (2025-07-28 01:53:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 493.035619ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 204.632318ms FAIL (2025-07-28 01:53:11 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 204.667076ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [........X.X....X.XX.....................X...................] duration: 432.684139ms FAIL (2025-07-28 01:52:55 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 432.717493ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 31 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.............................] duration: 13.884730692s FAIL (2025-07-28 01:52:29 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 13.884819745s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..................X..XX.X..X.X..............................] duration: 554.680029ms FAIL (2025-07-28 01:52:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 554.717759ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...................X..XX.X..X.X.............................] duration: 700.117577ms FAIL (2025-07-28 01:52:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 700.144812ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 220.568795ms FAIL (2025-07-28 01:52:01 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 220.595653ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........X.X....X.XX.....................X..................] duration: 549.66655ms FAIL (2025-07-28 01:51:56 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 549.704885ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 32 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX............................] duration: 7.630485625s FAIL (2025-07-28 01:51:18 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 7.630523465s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........X.X....X.XX.....................X.................] duration: 509.36498ms FAIL (2025-07-28 01:51:10 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 509.412047ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [....................X..XX.X..X.X............................] duration: 617.753311ms FAIL (2025-07-28 01:51:07 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 617.783919ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 33 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...........................] duration: 10.356654547s FAIL (2025-07-28 01:50:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 10.356700257s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 202.836065ms FAIL (2025-07-28 01:50:52 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 202.870792ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.....................X..XX.X..X.X...........................] duration: 536.411967ms FAIL (2025-07-28 01:50:28 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 536.447071ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 371.505274ms FAIL (2025-07-28 01:50:19 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 371.540342ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........X.X....X.XX.....................X................] duration: 420.715324ms FAIL (2025-07-28 01:50:18 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 420.751661ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [X...........X.X....X.XX.....................X...............] duration: 479.623275ms FAIL (2025-07-28 01:49:59 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 479.657727ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [......................X..XX.X..X.X..........................] duration: 393.219123ms FAIL (2025-07-28 01:49:47 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 393.300638ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 34 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX..........................] duration: 7.094867951s FAIL (2025-07-28 01:49:47 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 7.094900868s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 220.260801ms FAIL (2025-07-28 01:49:27 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 220.292473ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 35 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.........................] duration: 3.335945183s FAIL (2025-07-28 01:49:15 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 3.335980723s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 36 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX........................] duration: 7.933972944s FAIL (2025-07-28 01:48:56 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 7.934003638s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [.X...........X.X....X.XX.....................X..............] duration: 431.470603ms FAIL (2025-07-28 01:48:56 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 431.499607ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.......................X..XX.X..X.X.........................] duration: 423.11771ms FAIL (2025-07-28 01:48:44 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 423.154575ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 7 out of 60 is failed (threshold=3) [..X...........X.X....X.XX.....................X.............] duration: 456.374625ms FAIL (2025-07-28 01:48:36 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 456.403453ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [........................X..XX.X..X.X........................] duration: 611.279255ms FAIL (2025-07-28 01:48:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 611.306923ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [X............X..............................................] duration: 205.293196ms FAIL (2025-07-28 01:48:21 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 205.322984ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 37 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.......................] duration: 9.265852861s FAIL (2025-07-28 01:48:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 9.265886161s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.X............X.............................................] duration: 259.82228ms FAIL (2025-07-28 01:48:01 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 259.876114ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [X..X...........X.X....X.XX.....................X............] duration: 447.700761ms FAIL (2025-07-28 01:47:24 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 447.855411ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.........................X..XX.X..X.X.......................] duration: 394.286355ms FAIL (2025-07-28 01:47:23 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 394.329501ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX......................] duration: 3.207842523s FAIL (2025-07-28 01:46:55 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 3.207893419s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [..X............X............................................] duration: 195.732129ms FAIL (2025-07-28 01:46:53 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 195.768615ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [..........................X..XX.X..X.X......................] duration: 460.765639ms FAIL (2025-07-28 01:46:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 460.802459ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [...X............X...........................................] duration: 226.45548ms FAIL (2025-07-28 01:46:30 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 226.485954ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [.X..X...........X.X....X.XX.....................X...........] duration: 469.342899ms FAIL (2025-07-28 01:46:20 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 469.37694ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [...........................X..XX.X..X.X.....................] duration: 450.807665ms FAIL (2025-07-28 01:46:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 450.837129ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 39 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.....................] duration: 1.226962766s FAIL (2025-07-28 01:45:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1.227062117s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [....X............X..........................................] duration: 226.987708ms FAIL (2025-07-28 01:45:57 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 227.020281ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [..X..X...........X.X....X.XX.....................X..........] duration: 474.594784ms FAIL (2025-07-28 01:45:38 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 474.629487ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 40 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX....................] duration: 12.821122039s FAIL (2025-07-28 01:45:33 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 12.821151284s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 8 out of 60 is failed (threshold=3) [...X..X...........X.X....X.XX.....................X.........] duration: 424.810922ms FAIL (2025-07-28 01:45:01 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 424.844092ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [............................X..XX.X..X.X....................] duration: 439.783755ms FAIL (2025-07-28 01:44:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 439.815557ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 2 out of 60 is failed (threshold=3) [.....X............X.........................................] duration: 201.13777ms FAIL (2025-07-28 01:44:49 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/1K") Duration: 201.16805ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 41 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX...................] duration: 3.632011093s FAIL (2025-07-28 01:44:35 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 3.632041226s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)