Orbiter History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 452.289188ms FAIL (2025-07-28 02:21:17 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 452.324852ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..X.........................................................] duration: 627.525124ms FAIL (2025-07-28 02:20:23 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 627.636577ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...X........................................................] duration: 558.30635ms FAIL (2025-07-28 02:19:58 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 558.33645ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....X.......................................................] duration: 453.270503ms FAIL (2025-07-28 02:18:50 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 453.317903ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....X......................................................] duration: 460.098724ms FAIL (2025-07-28 02:18:17 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 460.15462ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [......X.....................................................] duration: 495.263924ms FAIL (2025-07-28 02:17:26 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 495.295279ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......X....................................................] duration: 482.974038ms FAIL (2025-07-28 02:16:27 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 483.006425ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 796.395566ms FAIL (2025-07-28 02:16:25 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 796.446987ms core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [........X...................................................] duration: 527.014131ms FAIL (2025-07-28 02:16:05 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 527.070634ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [X...........................................................] duration: 590.505358ms FAIL (2025-07-28 02:15:42 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 590.538114ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 2 out of 60 is failed (threshold=3) [XX..........................................................] duration: 5.579857169s FAIL (2025-07-28 02:15:19 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 5.579983842s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........X..................................................] duration: 634.499704ms FAIL (2025-07-28 02:15:19 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 634.539595ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.X..........................................................] duration: 518.481904ms FAIL (2025-07-28 02:15:08 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 518.518695ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........X.................................................] duration: 426.743249ms FAIL (2025-07-28 02:14:37 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 426.777158ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.X.........................................................] duration: 540.33393ms FAIL (2025-07-28 02:14:24 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 540.428989ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 3 out of 60 is failed (threshold=3) [XXX.........................................................] duration: 8.681683696s FAIL (2025-07-28 02:14:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 8.681718287s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...........X................................................] duration: 496.759189ms FAIL (2025-07-28 02:14:00 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 496.79402ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [.X.X........................................................] duration: 476.156997ms FAIL (2025-07-28 02:13:49 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 476.261338ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 4 out of 60 is failed (threshold=3) [XXXX........................................................] duration: 1.697913421s FAIL (2025-07-28 02:13:39 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1.69797089s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [............X...............................................] duration: 423.961677ms FAIL (2025-07-28 02:13:39 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 424.007894ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [XXXXX.......................................................] duration: 1.793607225s FAIL (2025-07-28 02:13:12 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1.793639697s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 2 out of 60 is failed (threshold=3) [..X.X.......................................................] duration: 428.68826ms FAIL (2025-07-28 02:12:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 428.716318ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.............X..............................................] duration: 481.939574ms FAIL (2025-07-28 02:12:46 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 481.972932ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..............X.............................................] duration: 606.894318ms FAIL (2025-07-28 02:12:28 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 606.935844ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [XXXXXX......................................................] duration: 3.074437329s FAIL (2025-07-28 02:12:10 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 3.074474725s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [X..X.X......................................................] duration: 469.089588ms FAIL (2025-07-28 02:11:51 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 469.127029ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...............X............................................] duration: 518.015825ms FAIL (2025-07-28 02:11:30 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 518.049035ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 7 out of 60 is failed (threshold=3) [XXXXXXX.....................................................] duration: 4.982875832s FAIL (2025-07-28 02:11:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 4.982914871s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [................X...........................................] duration: 517.222506ms FAIL (2025-07-28 02:11:00 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 517.259783ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 3 out of 60 is failed (threshold=3) [.X..X.X.....................................................] duration: 407.688872ms FAIL (2025-07-28 02:10:40 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 407.720952ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 8 out of 60 is failed (threshold=3) [XXXXXXXX....................................................] duration: 5.93682413s FAIL (2025-07-28 02:10:02 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 5.936860929s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.................X..........................................] duration: 376.654509ms FAIL (2025-07-28 02:09:56 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 376.681168ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 4 out of 60 is failed (threshold=3) [X.X..X.X....................................................] duration: 592.477734ms FAIL (2025-07-28 02:09:48 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 592.50911ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 397.338848ms FAIL (2025-07-28 02:09:20 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 397.366698ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 9 out of 60 is failed (threshold=3) [XXXXXXXXX...................................................] duration: 1.769161157s FAIL (2025-07-28 02:09:01 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1.769210872s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 552.267625ms FAIL (2025-07-28 02:08:58 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 552.30172ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [XX.X..X.X...................................................] duration: 462.42642ms FAIL (2025-07-28 02:08:56 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 462.460908ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 585.49032ms FAIL (2025-07-28 02:08:09 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 585.539609ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [.XX.X..X.X..................................................] duration: 623.937004ms FAIL (2025-07-28 02:08:04 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 623.989492ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 10 out of 60 is failed (threshold=3) [XXXXXXXXXX..................................................] duration: 7.336417815s FAIL (2025-07-28 02:07:55 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 7.336452076s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 5 out of 60 is failed (threshold=3) [..XX.X..X.X.................................................] duration: 453.440602ms FAIL (2025-07-28 02:07:33 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 453.476318ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 483.596969ms FAIL (2025-07-28 02:07:14 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 483.626244ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 11 out of 60 is failed (threshold=3) [XXXXXXXXXXX.................................................] duration: 1.093806362s FAIL (2025-07-28 02:06:36 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 1.093843888s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [X..XX.X..X.X................................................] duration: 523.369373ms FAIL (2025-07-28 02:06:20 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 523.421184ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 2 out of 60 is failed (threshold=3) [X.....................X.....................................] duration: 422.966477ms FAIL (2025-07-28 02:06:13 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 423.003159ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 12 out of 60 is failed (threshold=3) [XXXXXXXXXXXX................................................] duration: 5.208225163s FAIL (2025-07-28 02:05:59 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 5.208283744s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [XXXXXXXXXXXXX...............................................] duration: 2.83797084s FAIL (2025-07-28 02:05:27 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 2.838004762s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 3 out of 60 is failed (threshold=3) [XX.....................X....................................] duration: 564.378407ms FAIL (2025-07-28 02:05:22 UTC) orbiter warning: uplink: object not found ("orbiter_us_0/4M") Duration: 564.414343ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 6 out of 60 is failed (threshold=3) [.X..XX.X..X.X...............................................] duration: 600.96069ms FAIL (2025-07-28 02:05:14 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== , piece_id=PQWQOLWHA44KF6AHJ4BCL2567FZHP554GUYJGFV63RIHTFAP2GSA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: <nil> Jul 28 01:27:59.112933048 (27h36m14.893866424s) [noise=true , address=104.36.82.7:4911 , piece_id=HDVASLJGDJQQU37PBWK3M3ACJY7GZMVESDMC5ES2G6UIWCLGHHRA, seg_pos={0 0}, upload=26326]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu): protocol: context canceled; context canceled Jul 28 01:27:59.112973262 (27h36m14.893906648s) [seg_pos={0 0}, upload=26326]: Done waiting for segment. successful: 15 optimal: 13 errs: <nil> Jul 28 01:27:59.113155226 (27h36m14.894088593s) [upload=26326]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 28 01:27:59.113164520 (27h36m14.894097903s) [upload=26326]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 28 01:27:59.264196069 (27h36m15.045129437s) [upload=26326]: Done uploading segments... Jul 28 01:27:59.264214498 (27h36m15.045147862s) [upload=26326]: Upload finished. err: <nil> Jul 28 01:28:45.902335981 (27h37m1.683269346s) [upload=26340]: Starting upload Jul 28 01:28:45.902447669 (27h37m1.683381054s) [upload=26340]: Done starting upload Jul 28 01:28:45.902546661 (27h37m1.683480033s) [upload=26340]: Uploading segments... Jul 28 01:28:45.902825526 (27h37m1.683758908s) [upload=26340]: Got next segment. Inline: false Jul 28 01:28:45.902866565 (27h37m1.683799933s) [upload=26340]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 28 01:28:45.902874066 (27h37m1.683807435s) [upload=26340]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 28 01:28:45.942346355 (27h37m1.723279721s) [seg_pos={0 0}, upload=26340]: Begin upload segment... Jul 28 01:28:45.942384932 (27h37m1.723318299s) [seg_pos={0 0}, upload=26340]: Done begin upload segment. Jul 28 01:28:45.942407919 (27h37m1.723341298s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.942846704 (27h37m1.723780089s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943144797 (27h37m1.724078182s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943276384 (27h37m1.724209769s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943679220 (27h37m1.724612605s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.943736694 (27h37m1.724670101s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944214415 (27h37m1.725147799s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944434175 (27h37m1.725367558s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944594921 (27h37m1.725528313s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.944891709 (27h37m1.725825092s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.947396208 (27h37m1.728329595s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.948840275 (27h37m1.729773667s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.950540776 (27h37m1.731474164s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952210949 (27h37m1.733144342s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.952753847 (27h37m1.73368725s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:45.953224449 (27h37m1.734157852s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Uploading piece... Jul 28 01:28:46.061971380 (27h37m1.842904749s) [upload=26340]: Next returned nil segment Jul 28 01:28:46.061989780 (27h37m1.842923145s) [upload=26340]: Waiting for error group managing segments... Jul 28 01:28:46.077809671 (27h37m1.858743034s) [noise=true , address=45.91.116.10:4082 , piece_id=ECKSG7KSGT67S6DJ2HX6QB7LWVNWRKER3INGF7MLBGMV5XFWTKFA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078142910 (27h37m1.859076271s) [noise=true , address=45.91.116.6:4973 , piece_id=WIQEASIFIE6HYVOWVMEQHL6AXNY7ODKR2FXZM77CP7CNUS22477A, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078586142 (27h37m1.859519513s) [noise=true , address=77.90.31.14:4663 , piece_id=3MA4JBTLBECNLLXXHR2JFRXSPZJYDJJAH7ZQ5UGA7LVBVK6IAXGQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.078917574 (27h37m1.85985095s) [noise=true , address=77.90.31.10:4517 , piece_id=ZIHCMPYYINE6ZMKEWKMDQY7PCGC2QE6OWKIINDPEOA3U4RL47EEQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079304960 (27h37m1.860238326s) [noise=true , address=45.91.116.10:4081 , piece_id=V6DCPAU3Y6XBKGTU7QFJZI3GJJ6EQT3TLOKTLI2NOECBHC5RO5RA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079718869 (27h37m1.860652294s) [noise=true , address=45.91.116.9:4262 , piece_id=GFDBWA47TQXCU6STMRC74BMRKENMK2AFKRTNS5CZGR4HM46WUATQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.079882565 (27h37m1.860815931s) [noise=true , address=45.91.116.5:4298 , piece_id=UHEXPYNKZMRQHH36HVNPVP6U62T57XYYLVLC2VAELPULT6MXJEPA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080612161 (27h37m1.861545541s) [noise=true , address=77.90.31.15:4699 , piece_id=J52R42HQH72VUEV2EO4YEVML3CM4JM4PJQ3U2QF7QIBTC5SAKVNQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.080638270 (27h37m1.86157164s) [noise=true , address=77.90.31.12:4590 , piece_id=3MQPITJMUXZLPODFKGQOPZEQKVHKLIIYTV6JNYCT3D7V527WMYJQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.081099103 (27h37m1.86203247s) [noise=true , address=77.90.31.11:4553 , piece_id=GNBKEQKZQC3OG23RJIMDDHU7C3AMDN3RM52JKO4AOAVBVGRU2JAA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:28:46.083317151 (27h37m1.864250521s) [noise=true , address=77.90.31.13:4627 , piece_id=V63XZY27FJHWJCRY4K6FMRWOKDRXWR4H3UH6O42BZFSJVFN3UBXA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:10.869770707 (27h37m26.650704082s) [noise=true , address=104.36.82.2:4730 , piece_id=FTRRQPDXNGRW4PI6GS66V6DBUOIGGVBBPHKMHXD5BRI7LWC3DCQA, seg_pos={0 0}, upload=26340]: Done uploading piece. err: <nil> Jul 28 01:29:15.868233676 (27h37m31.649167041s) [noise=true , address=104.36.82.9:4007 , piece_id=7YXDGNYUG7EWQI735WHOBYP62D2FWRJHZMGXQJK2NXTBT32NP4DQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868322503 (27h37m31.649255873s) [noise=true , address=104.36.82.6:4876 , piece_id=XPGO5GDN6UP4MJA3UO7SA75LOE56NHDMBLMLWNASATUQAS6X67UQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:1jMZpvqeRXcQPi1XvVQpVpzU7ZkiemTZCE1yvfXDTjG4NEsmpP, address:104.36.82.6:4876): protocol: read tcp 159.223.143.39:55698->104.36.82.6:4876: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:15.868398743 (27h37m31.649332117s) [noise=true , address=104.36.82.14:4221 , piece_id=HJMTKTPB3ZBSOGXDKTYVWW3PKIUNX6U2CIZOMQWP6BXOEYBX2OYQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868516683 (27h37m31.649450048s) [noise=true , address=104.36.82.15:4258 , piece_id=JITWLI2PLAUW24G2GOK4S45KWGJHY526GP2I3I75SKWEKKVF6WUQ, seg_pos={0 0}, upload=26340]: Done uploading piece. err: ecclient: upload failed (node:12G8RkyjiDQ2nNkM8drWhAKuBQfGizv711hyBDNB11noDzq1xtF, address:104.36.82.15:4258): protocol: context deadline exceeded; context deadline exceeded Jul 28 01:29:15.868532947 (27h37m31.649466317s) [seg_pos={0 0}, upload=26340]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 28 01:29:15.905035802 (27h37m31.685969211s) [upload=26340]: Done uploading segments... Jul 28 01:29:15.905065282 (27h37m31.685998682s) [upload=26340]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 600.993026ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [XXXXXXXXXXXXXX..............................................] duration: 7.649259328s FAIL (2025-07-28 02:05:00 UTC) orbiter warning: ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 6) ======LOGS====== F3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882285064 (27h36m28.663218443s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882585344 (27h36m28.663518719s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882804243 (27h36m28.663737629s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.882937735 (27h36m28.66387112s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883349288 (27h36m28.664282672s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883424548 (27h36m28.66435793s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883665425 (27h36m28.664598805s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.883901205 (27h36m28.66483459s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.884605017 (27h36m28.665538403s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887590122 (27h36m28.668523504s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.887994252 (27h36m28.668927639s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888318879 (27h36m28.669252263s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.888917492 (27h36m28.669850877s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Uploading piece... Jul 28 01:28:12.963939288 (27h36m28.744872658s) [upload=26331]: Next returned nil segment Jul 28 01:28:12.963954855 (27h36m28.744888232s) [upload=26331]: Waiting for error group managing segments... Jul 28 01:28:13.068031813 (27h36m28.848965184s) [noise=true , address=45.91.116.8:4062 , piece_id=4QUSPHKKJKCVQZ53KAWQO3UGQ5J64XPOORLJKXKETUHBLBPIGW5Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:13.528059962 (27h36m29.308993324s) [noise=true , address=64.49.13.138:4482 , piece_id=QLWNKGQDA5INQGNCUQ4M5VE7R2V7OA5P4I2FUDELRZCKJU2K36MA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:14.222309126 (27h36m30.003242496s) [noise=true , address=79.127.201.215:4345 , piece_id=4NHXVV7FX65YOKUL7EUBA6LOZYPNUQSRB45K6KHDCGPH6QL56MPA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:28:31.557701474 (27h36m47.338634842s) [noise=true , address=104.36.82.19:4414 , piece_id=74KLOHLUHA32T5ZUKMA4237CRDJORKXTEVDZAUBSTXPV7SCPZXDQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:01.076741528 (27h37m16.857674907s) [noise=true , address=104.36.82.43:4864 , piece_id=PRRWIQAK4MWDAXOTODF5KL4EZ2H5G6DI4IKEIWSIDD7YFGPWBOFA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:06.255469463 (27h37m22.036402837s) [noise=true , address=104.36.82.38:4700 , piece_id=SRM3CS4MCR4JCHYGLWPSTUQGV3L4TA6DSBPJF2B5V5ZKMO2UN6AQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: <nil> Jul 28 01:29:12.811812696 (27h37m28.592746066s) [noise=true , address=104.36.82.21:4504 , piece_id=HALPL4LEUBJ7P7Q3AWOH4KHGW4CUFWIESX5GTQNB5WCZXVA33PSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1EKUezs16ty651Y224ZZugySQQBQjEYRnRghbjdX8uFQBor2hM, address:104.36.82.21:4504): protocol: rpc: dial tcp 104.36.82.21:4504: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811902114 (27h37m28.592835497s) [noise=true , address=104.36.82.42:4830 , piece_id=MM4GF3M3T52FBJWN5QZY2HLYWEGIUNXXOZ5LI4ETYTLPA7QMSTQA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:142ocSHHtv8BpSebXFNk5qk1Y23a8pbFAZqE3cq6i3RMX3dPw9, address:104.36.82.42:4830): protocol: rpc: dial tcp 104.36.82.42:4830: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.811970299 (27h37m28.592903675s) [noise=true , address=104.36.82.13:4207 , piece_id=JLW2FUQK74GY44YEMKCMTL2XPRLMLXOJCXZJRVYBBIQQ4UJL2WSA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1bS1oz1rMxh8k1ZN4PALDgWTWw61uGGiqDX3gbXq49ehbDSFjw, address:104.36.82.13:4207): protocol: rpc: dial tcp 104.36.82.13:4207: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812029358 (27h37m28.592962739s) [noise=true , address=104.36.82.40:4967 , piece_id=UFTYXVB4BFW3NDXONKLTG52S6PLIXMEL4GX3NC2MFFYK4V3K2DNA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: rpc: dial tcp 104.36.82.40:4967: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812231923 (27h37m28.593165291s) [noise=true , address=104.36.82.50:4037 , piece_id=LZMKLYRW4QVKZNTELWQSXX3ZMWFHXF5M2SESMBCWWZEYGLRT5DJA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:14gJ4XfdRiFLVEixdjSP5QB8F9toGcb6xoxP7kXxX1vXJm5TDg, address:104.36.82.50:4037): protocol: rpc: dial tcp 104.36.82.50:4037: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812293794 (27h37m28.593227161s) [noise=true , address=104.36.82.56:4264 , piece_id=LC5K5SUBBN4CIQ3QLTYTTPDK77TF5DHCNYFXEMLQED3PVMCK7VBA, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12w76j1GsXyDuZPGEbpgG6o38ZNsqYRDbyZ4qr8ybqZYipEYJWk, address:104.36.82.56:4264): protocol: rpc: dial tcp 104.36.82.56:4264: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812334574 (27h37m28.59326795s) [noise=true , address=104.36.82.12:4154 , piece_id=DVZ6Y6IQ23PMOPKLQQTXGBKQ2QB4GNVJP4MXCDKHYQF4AGSGYF7A, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12S7Xsfnti3QuD9apBq1sgSjmahnAYA6LYbXqyckbLY4UU84ZQF, address:104.36.82.12:4154): protocol: rpc: dial tcp 104.36.82.12:4154: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812396992 (27h37m28.593330371s) [noise=true , address=104.36.82.47:4545 , piece_id=5UQVRPJ7D4SU6R54DMSZIFRIOQONC2CCCVW3MELBSS4YZTASTJSQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12euvmhGegnPYufoKKQ1on9FTv2AhuicSVYuWyXtDiY9BLdDMYE, address:104.36.82.47:4545): protocol: rpc: dial tcp 104.36.82.47:4545: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812418145 (27h37m28.593351526s) [noise=true , address=104.36.82.55:4207 , piece_id=J3EM3WIGVIVCXJPNK6ASCRZP5APC4EQTUWJM6AZR4DYTUITO3AJQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:46424->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812431193 (27h37m28.593364577s) [noise=true , address=104.36.82.41:4770 , piece_id=SQ7XCUV4A4BNNKDEED2PBJI6XQ2YXON2RUOSNI6T4MJTRN4PIPKQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1Vpr5UhYBKjdwam6GJFcMqrwjdkaCstSTrxgX8AzX5VjhWotK5, address:104.36.82.41:4770): protocol: rpc: dial tcp 104.36.82.41:4770: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812442260 (27h37m28.593375629s) [noise=true , address=104.36.82.54:4169 , piece_id=N5KI35YSIIQCGOY3M7CIGRI42NEMEWJQCBGJVQBPHEGYG6BXME4Q, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:1eSyfixGcCi7TCFscr473WQvg7ujGfcr33pUdGN4dRngroiN5i, address:104.36.82.54:4169): protocol: rpc: dial tcp 104.36.82.54:4169: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812461361 (27h37m28.59339474s) [noise=true , address=104.36.82.9:4009 , piece_id=NZRQLKHMSYHBXZ43AM5XHCEWVLXX33KT4JIL7Z5DWWVKP64EMNEQ, seg_pos={0 0}, upload=26331]: Done uploading piece. err: ecclient: upload failed (node:12n33Xie5Hk1vaKc279HvGnL2QBEYsNzieL2grpnANJpy7Qu2wb, address:104.36.82.9:4009): protocol: rpc: dial tcp 104.36.82.9:4009: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 28 01:29:12.812499178 (27h37m28.593432561s) [seg_pos={0 0}, upload=26331]: Done waiting for segment. successful: 6 optimal: 14 errs: <nil> Jul 28 01:29:12.837163371 (27h37m28.618096739s) [upload=26331]: Done uploading segments... Jul 28 01:29:12.837190397 (27h37m28.618123771s) [upload=26331]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 6) Duration: 7.649290606s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)