Orbiter History

checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [...................................X........X.X.X...XX......] duration: 19.370492542s FAIL (2025-07-26 22:35:40 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 19.370515065s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..................................................X.........] duration: 215.329677ms FAIL (2025-07-26 22:35:38 UTC) orbiter warning: uplink: too many requests Duration: 215.367725ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 39 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..X] duration: 1m0.041840454s FAIL (2025-07-26 22:35:27 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== 661]: Got next segment. Inline: false Jul 26 22:34:27.951601133 (42m43.732534511s) [upload=661]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:34:27.951620077 (42m43.732553458s) [upload=661]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:34:27.987616768 (42m43.768550155s) [seg_pos={0 0}, upload=661]: Begin upload segment... Jul 26 22:34:27.987677035 (42m43.768610417s) [seg_pos={0 0}, upload=661]: Done begin upload segment. Jul 26 22:34:27.987710731 (42m43.76864416s) [noise=true , address=64.49.13.140:4516 , piece_id=VNEH2NXYNH4HC6UNLLIA4VI56ORP57CGDIVPR7BFONOLDIY7H6PA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.988347840 (42m43.769281227s) [noise=true , address=79.127.205.246:4288 , piece_id=VHIKAX2FZBGPSFRZMAWF6J27NHRBL6TSHE5FZMET6DXKXLWUUWUA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.988766163 (42m43.769699552s) [noise=true , address=79.127.219.51:4212 , piece_id=LYUAPOLRDGBWIDGMDX6LI4X2LZZP2QHOV2TOBEZHEGXKNZUZW3HA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.989132146 (42m43.770065535s) [noise=true , address=45.91.116.13:4175 , piece_id=ZLRYM2YSOEZ5NWSOWK5BPKYQIANMVJVSGP37DT64E4VUO7P6HTDQ, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.989479830 (42m43.770413219s) [noise=true , address=79.127.201.215:4347 , piece_id=XCKRUQYGDAUMHA22VBVP64UCZISA2GDA5PYW6LGFCB7I6577KXUQ, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.989836216 (42m43.770769605s) [noise=true , address=77.90.31.19:4871 , piece_id=QVCO3O6M37CI6BJOSOPAIAMIAUFXB2RGZNYPK3TBAZEADAKETM2A, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.990228021 (42m43.77116141s) [noise=true , address=79.127.219.47:4069 , piece_id=4THURBUEKRP3JM7QZCWVRE5Y5ILP3GI26LA7X7QODSSJ4PNYY6EA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.993010821 (42m43.773944208s) [noise=true , address=77.90.31.17:4774 , piece_id=QG6LKYVX23N5L4CQFZ6IMRVOTLACEZUIOTBFOAZSLJFTEVLSRDIQ, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.993550663 (42m43.774484051s) [noise=true , address=77.90.31.15:4713 , piece_id=7ONT4B7UNK53T4LYXCAXK6NBUQGN45Z27DYZZYU4AI4QHDKRXS3A, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.994855210 (42m43.7757886s) [noise=true , address=79.127.220.97:4361 , piece_id=2FH5LIZV2S5RAHNKOUOZMOGXA5VVGYIADMKQFTHSDYLFOVJNAD2Q, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.995342767 (42m43.776276154s) [noise=true , address=45.91.116.9:4267 , piece_id=2VHHVANAMTK6KINVQVVEPIUUTKJL3KJABZROESEE2SOLTMSBL3RQ, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.995778002 (42m43.776711391s) [noise=true , address=64.49.13.138:4482 , piece_id=NM5Z2MIKYTIP7XUEKOGUM4WVBYY5ZJZDMTSW4YV2BNHYA7VB6R7A, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.996318448 (42m43.777251835s) [noise=true , address=79.127.219.38:4003 , piece_id=T3HRQHU4G34QQPLHXBXW5IMJTFKO2BWS4WSWRJKGO3M6D3FSW7ZA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.996711652 (42m43.777645038s) [noise=true , address=79.127.205.234:4612 , piece_id=PCVIO3LMYYN4TINAB5DVUS47NCHCRKHQOMQWMQIH26H4SEJQSC5A, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.997100832 (42m43.778034226s) [noise=true , address=79.127.205.245:4237 , piece_id=GEJFAX5OPXASZESLJLRPOVE5W6LLUFHDP2KIO7FK4U37KY5T6FIA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.997489534 (42m43.778422924s) [noise=true , address=45.91.116.8:4065 , piece_id=X255M7QFAT5JOPBZO3DYBNHOKDKSSBODKMKXF7ZKVDSW7EW7TGHQ, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.997921939 (42m43.77885533s) [noise=true , address=77.90.31.12:4608 , piece_id=QCZE7AFBCTBJCNAVR67KQSU3FRNVNQQQTFP3LQ4QQRHBIKWCXJLQ, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:27.998345471 (42m43.77927886s) [noise=true , address=77.90.31.18:4817 , piece_id=NSAQ4EDSLCYZSSN7TB4HEQ5XXVVH6QSPHXJKIDI36BITOYYMVXXA, seg_pos={0 0}, upload=661]: Uploading piece... Jul 26 22:34:28.056033124 (42m43.836966506s) [noise=true , address=79.127.219.47:4069 , piece_id=4THURBUEKRP3JM7QZCWVRE5Y5ILP3GI26LA7X7QODSSJ4PNYY6EA, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:18dK6RxbMxU4a46pW4YPiQW87xgpeoTb1ouqE9fzTdzTAFAa3W, address:79.127.219.47:4069): protocol: storage node overloaded, request limit: 2000 Jul 26 22:34:28.095674502 (42m43.876607888s) [upload=661]: Next returned nil segment Jul 26 22:34:28.095713720 (42m43.876647086s) [upload=661]: Waiting for error group managing segments... Jul 26 22:34:28.101217341 (42m43.882150718s) [noise=true , address=79.127.219.38:4003 , piece_id=T3HRQHU4G34QQPLHXBXW5IMJTFKO2BWS4WSWRJKGO3M6D3FSW7ZA, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:28.119364618 (42m43.900298008s) [noise=true , address=79.127.219.51:4212 , piece_id=LYUAPOLRDGBWIDGMDX6LI4X2LZZP2QHOV2TOBEZHEGXKNZUZW3HA, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:17d1EWYAhx1xHZYRMKakqNkNpUAEGFG7HWcsfqGYvb3m1jyGFL, address:79.127.219.51:4212): protocol: storage node overloaded, request limit: 2000 Jul 26 22:34:28.138620488 (42m43.919553877s) [noise=true , address=79.127.220.97:4361 , piece_id=2FH5LIZV2S5RAHNKOUOZMOGXA5VVGYIADMKQFTHSDYLFOVJNAD2Q, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:1ojv5zf5QshKyMqGypaEaYmGvzt96UM9kZgyGeFVPMFk8Twzyn, address:79.127.220.97:4361): protocol: storage node overloaded, request limit: 2000 Jul 26 22:34:28.144505433 (42m43.92543881s) [noise=true , address=77.90.31.17:4774 , piece_id=QG6LKYVX23N5L4CQFZ6IMRVOTLACEZUIOTBFOAZSLJFTEVLSRDIQ, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:28.146709992 (42m43.927643371s) [noise=true , address=77.90.31.12:4608 , piece_id=QCZE7AFBCTBJCNAVR67KQSU3FRNVNQQQTFP3LQ4QQRHBIKWCXJLQ, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:28.631921429 (42m44.412854796s) [noise=true , address=79.127.205.246:4288 , piece_id=VHIKAX2FZBGPSFRZMAWF6J27NHRBL6TSHE5FZMET6DXKXLWUUWUA, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:28.678668075 (42m44.459601453s) [noise=true , address=79.127.205.234:4612 , piece_id=PCVIO3LMYYN4TINAB5DVUS47NCHCRKHQOMQWMQIH26H4SEJQSC5A, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:29.137248803 (42m44.918182166s) [noise=true , address=79.127.205.245:4237 , piece_id=GEJFAX5OPXASZESLJLRPOVE5W6LLUFHDP2KIO7FK4U37KY5T6FIA, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:31.771346822 (42m47.552280203s) [noise=true , address=45.91.116.13:4175 , piece_id=ZLRYM2YSOEZ5NWSOWK5BPKYQIANMVJVSGP37DT64E4VUO7P6HTDQ, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:124YLi7uuYNfk8TYrgdAkmrGnKmCfRC1MBmeJypA2hwHyWdp43U, address:45.91.116.13:4175): protocol: storage node overloaded, request limit: 2000 Jul 26 22:34:32.565641205 (42m48.346574594s) [noise=true , address=45.91.116.8:4065 , piece_id=X255M7QFAT5JOPBZO3DYBNHOKDKSSBODKMKXF7ZKVDSW7EW7TGHQ, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:17EPCQmYb816aegE9vPSoJYByMUEvhP4DUiabis2gh9QjXQC3s, address:45.91.116.8:4065): protocol: storage node overloaded, request limit: 2000 Jul 26 22:34:33.360957537 (42m49.141890915s) [noise=true , address=79.127.201.215:4347 , piece_id=XCKRUQYGDAUMHA22VBVP64UCZISA2GDA5PYW6LGFCB7I6577KXUQ, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:34.784741480 (42m50.565674868s) [noise=true , address=45.91.116.9:4267 , piece_id=2VHHVANAMTK6KINVQVVEPIUUTKJL3KJABZROESEE2SOLTMSBL3RQ, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:35.843967454 (42m51.624900832s) [noise=true , address=64.49.13.138:4482 , piece_id=NM5Z2MIKYTIP7XUEKOGUM4WVBYY5ZJZDMTSW4YV2BNHYA7VB6R7A, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:34:36.075665804 (42m51.85659917s) [noise=true , address=64.49.13.140:4516 , piece_id=VNEH2NXYNH4HC6UNLLIA4VI56ORP57CGDIVPR7BFONOLDIY7H6PA, seg_pos={0 0}, upload=661]: Done uploading piece. err: <nil> Jul 26 22:35:27.909515279 (43m43.690448663s) [noise=true , address=77.90.31.19:4871 , piece_id=QVCO3O6M37CI6BJOSOPAIAMIAUFXB2RGZNYPK3TBAZEADAKETM2A, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:12hrM59APBvzYUrwWMvSUQVMSkzNPxaNFDWEZn2J96xYTdv4Ezm, address:77.90.31.19:4871): protocol: rpc: dial tcp 77.90.31.19:4871: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:35:27.909647421 (43m43.690580801s) [noise=true , address=77.90.31.15:4713 , piece_id=7ONT4B7UNK53T4LYXCAXK6NBUQGN45Z27DYZZYU4AI4QHDKRXS3A, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:1Liv9RcEgfjxodPuadijzkcUGgJEy7LfKqHGFhaJCKbcxKkNN8, address:77.90.31.15:4713): protocol: write tcp 159.223.143.39:53460->77.90.31.15:4713: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:35:27.909740858 (43m43.690674235s) [noise=true , address=77.90.31.18:4817 , piece_id=NSAQ4EDSLCYZSSN7TB4HEQ5XXVVH6QSPHXJKIDI36BITOYYMVXXA, seg_pos={0 0}, upload=661]: Done uploading piece. err: ecclient: upload failed (node:12Qv5LBuANP9vS2vbsVNrJ7Mkc3SqXwTNCBzLvN955RHFKQVv5x, address:77.90.31.18:4817): protocol: write tcp 159.223.143.39:56042->77.90.31.18:4817: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:35:27.909776921 (43m43.690710298s) [seg_pos={0 0}, upload=661]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 22:35:27.947311027 (43m43.728244408s) [upload=661]: Done uploading segments... Jul 26 22:35:27.947342152 (43m43.72827553s) [upload=661]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.041877536s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..................X.........................................] duration: 489.097538ms FAIL (2025-07-26 22:35:25 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 489.131552ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [X..X........XXX.XXX.X.XX..X..X......X.............X.........] duration: 441.152948ms FAIL (2025-07-26 22:35:16 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 441.184713ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [...................................................X........] duration: 357.16577ms FAIL (2025-07-26 22:34:53 UTC) orbiter warning: uplink: too many requests Duration: 357.196019ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [........XX..XXX..X..XX..X..XX.X....X....X........X..........] duration: 417.635535ms FAIL (2025-07-26 22:34:51 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 417.78023ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [...................X........................................] duration: 522.967641ms FAIL (2025-07-26 22:34:50 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 522.990313ms edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [....................................X........X.X.X...XX.....] duration: 1.53728997s FAIL (2025-07-26 22:34:50 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.538325747s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [....................X.......................................] duration: 750.789941ms FAIL (2025-07-26 22:34:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 750.847867ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [.X..X........XXX.XXX.X.XX..X..X......X.............X........] duration: 493.470179ms FAIL (2025-07-26 22:34:12 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 493.499066ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.........XX..XXX..X..XX..X..XX.X....X....X........X.........] duration: 717.26462ms FAIL (2025-07-26 22:34:09 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 717.30003ms core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.....................................X........X.X.X...XX....] duration: 1.940973544s FAIL (2025-07-26 22:34:09 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.94101523s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX..] duration: 33.715051763s FAIL (2025-07-26 22:33:54 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0200 (39m38.953613576s) [upload=602]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 22:31:25.092817481 (39m40.873750866s) [upload=615]: Starting upload Jul 26 22:31:25.092917455 (39m40.873850841s) [upload=615]: Done starting upload Jul 26 22:31:25.093095667 (39m40.874029048s) [upload=615]: Uploading segments... Jul 26 22:31:25.093538635 (39m40.874472036s) [upload=615]: Got next segment. Inline: false Jul 26 22:31:25.093618508 (39m40.874551888s) [upload=615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:31:25.093624913 (39m40.874558297s) [upload=615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:31:25.131355162 (39m40.912288549s) [seg_pos={0 0}, upload=615]: Begin upload segment... Jul 26 22:31:25.131427766 (39m40.912361148s) [seg_pos={0 0}, upload=615]: Done begin upload segment. Jul 26 22:31:25.131463629 (39m40.912397017s) [noise=true , address=64.49.13.138:4482 , piece_id=DLWMFB534AQGVWQCALOCRHHIVQIORWED7G3YX3OPDLKK45ZYL7WQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.131879895 (39m40.91281328s) [noise=true , address=79.127.220.101:4072 , piece_id=XMWGQ3KT4HPYEXGLM4UVSGD24TRWC2XYIES2CF6SBI55CDACYAGA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.132543852 (39m40.91347724s) [noise=true , address=79.127.201.215:4351 , piece_id=TMNXYMBDACXBGB66DXK5FFEGCERRNY3JUSAHOAHOHPMZ75V5J6MQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.132579299 (39m40.913512688s) [noise=true , address=45.91.116.9:4269 , piece_id=MFXVLH73CJA24KMLKCVNJCZNKMEPPTT4OJRS467HSSAZ4EWEN3QA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133014796 (39m40.913948163s) [noise=true , address=79.127.205.246:4266 , piece_id=R6FB6SWR2WOUW6QTVYGU6VWEX3DPMFKFEXEKBNUM22KVYCIXZAAA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133072582 (39m40.914005971s) [noise=true , address=77.90.31.18:4822 , piece_id=BU7KV4R7HN5UTIQM5ZB5FSK53PWBDIWMZHWSS5YJFG347OGSTHZA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133510863 (39m40.914444253s) [noise=true , address=64.49.13.155:4918 , piece_id=62RZCYTG4WBNYDJBNJYI5ALYR32AEF4VLLL3UHKGT23AHU4CANVQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133529028 (39m40.914462418s) [noise=true , address=79.127.205.233:4656 , piece_id=M55U3Z7Q3IVZCBVMQOUNVLJOPPCQXDP3WJW4GS2BJB5JGK6YXEAQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133983589 (39m40.914917048s) [noise=true , address=79.127.205.234:4605 , piece_id=5SKQOIOWVTGSFTQ5UZBYFZUEMY6U5Q5HANHREPXKDUVUIGZJBQXQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134054538 (39m40.914987927s) [noise=true , address=79.127.205.235:4682 , piece_id=5T24674RXWOK7N5L6X75FEWYXUQGR5CEYZI4UUXT6OX624E5BXZQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134498895 (39m40.915432285s) [noise=true , address=77.90.31.21:4885 , piece_id=HYKGBHOAHXQDZBFQEURWPD7ANB2QPGY5DSDYCMFRLHJ4DSJAFQTQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134590489 (39m40.915523878s) [noise=true , address=79.127.219.48:4090 , piece_id=OCOSOC4AD3BXG7MR35TUXI4AXVPQXFHWYXC3QMLJZIBYM4UEUDVQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134966793 (39m40.915900183s) [noise=true , address=64.49.13.170:4721 , piece_id=5YM7KUCLHAYUDAJAIVORZ6QXDJOIH27NGKHOKBHYUW6ST6JRNLGA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135163729 (39m40.916097118s) [noise=true , address=156.146.43.225:4448 , piece_id=JCNIQ7BUJOVZN37PZHPCH7AKAKB5ONIIGB7BKXZRGVIPLMWFLWLQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135677391 (39m40.916610776s) [noise=true , address=79.127.219.38:4007 , piece_id=6ED5TDNA6C3CW3EOYUUWXSAZPAFKUAYOMJDNAYZXXL2HPAKCO2HA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135741108 (39m40.916674484s) [noise=true , address=45.91.116.11:4146 , piece_id=IFEFOFUQAR75XDLS5IVES5SDIIMGAT6R6ZVFH7XFQOUORMBOLRIA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.136284624 (39m40.917218003s) [noise=true , address=77.90.31.12:4608 , piece_id=RJ6ARFV3MTGZ34ELJPCKGVYHFNMKXJEVCT7JFF2CTMK5FMWXE5PQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.137524409 (39m40.918457794s) [noise=true , address=45.91.116.14:4222 , piece_id=BYVM6RJ4T3RQMJYVAROHAJCJX7MKGXJWN4O2B24ZR4GCN2PJ4EFQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.240969311 (39m41.021902693s) [upload=615]: Next returned nil segment Jul 26 22:31:25.240995734 (39m41.021929111s) [upload=615]: Waiting for error group managing segments... Jul 26 22:31:25.243994267 (39m41.024927648s) [noise=true , address=79.127.220.101:4072 , piece_id=XMWGQ3KT4HPYEXGLM4UVSGD24TRWC2XYIES2CF6SBI55CDACYAGA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12bQXjK4bv5583RV4BtjhC8WTDJgz255Ptvs6f5kqvJu4SA99mc, address:79.127.220.101:4072): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.245375223 (39m41.026308585s) [noise=true , address=79.127.219.38:4007 , piece_id=6ED5TDNA6C3CW3EOYUUWXSAZPAFKUAYOMJDNAYZXXL2HPAKCO2HA, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.291788207 (39m41.07272157s) [noise=true , address=77.90.31.12:4608 , piece_id=RJ6ARFV3MTGZ34ELJPCKGVYHFNMKXJEVCT7JFF2CTMK5FMWXE5PQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.388770962 (39m41.169704331s) [noise=true , address=45.91.116.11:4146 , piece_id=IFEFOFUQAR75XDLS5IVES5SDIIMGAT6R6ZVFH7XFQOUORMBOLRIA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.516260375 (39m41.297193757s) [noise=true , address=64.49.13.138:4482 , piece_id=DLWMFB534AQGVWQCALOCRHHIVQIORWED7G3YX3OPDLKK45ZYL7WQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.723538202 (39m41.504471568s) [noise=true , address=79.127.205.233:4656 , piece_id=M55U3Z7Q3IVZCBVMQOUNVLJOPPCQXDP3WJW4GS2BJB5JGK6YXEAQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.725133974 (39m41.506067342s) [noise=true , address=79.127.205.235:4682 , piece_id=5T24674RXWOK7N5L6X75FEWYXUQGR5CEYZI4UUXT6OX624E5BXZQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.748427597 (39m41.529360974s) [noise=true , address=79.127.205.246:4266 , piece_id=R6FB6SWR2WOUW6QTVYGU6VWEX3DPMFKFEXEKBNUM22KVYCIXZAAA, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.786472821 (39m41.567406198s) [noise=true , address=79.127.205.234:4605 , piece_id=5SKQOIOWVTGSFTQ5UZBYFZUEMY6U5Q5HANHREPXKDUVUIGZJBQXQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:26.151056459 (39m41.931989836s) [noise=true , address=45.91.116.9:4269 , piece_id=MFXVLH73CJA24KMLKCVNJCZNKMEPPTT4OJRS467HSSAZ4EWEN3QA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:26.431220045 (39m42.212153426s) [noise=true , address=45.91.116.14:4222 , piece_id=BYVM6RJ4T3RQMJYVAROHAJCJX7MKGXJWN4O2B24ZR4GCN2PJ4EFQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:26.952353227 (39m42.733286605s) [noise=true , address=64.49.13.170:4721 , piece_id=5YM7KUCLHAYUDAJAIVORZ6QXDJOIH27NGKHOKBHYUW6ST6JRNLGA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:27.155824153 (39m42.936757545s) [noise=true , address=156.146.43.225:4448 , piece_id=JCNIQ7BUJOVZN37PZHPCH7AKAKB5ONIIGB7BKXZRGVIPLMWFLWLQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:27.445015172 (39m43.225948552s) [noise=true , address=79.127.219.48:4090 , piece_id=OCOSOC4AD3BXG7MR35TUXI4AXVPQXFHWYXC3QMLJZIBYM4UEUDVQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:34.971697059 (39m50.752630428s) [noise=true , address=64.49.13.155:4918 , piece_id=62RZCYTG4WBNYDJBNJYI5ALYR32AEF4VLLL3UHKGT23AHU4CANVQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:42.955261339 (39m58.73619492s) [noise=true , address=77.90.31.21:4885 , piece_id=HYKGBHOAHXQDZBFQEURWPD7ANB2QPGY5DSDYCMFRLHJ4DSJAFQTQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:51.785580862 (40m7.566514236s) [noise=true , address=79.127.201.215:4351 , piece_id=TMNXYMBDACXBGB66DXK5FFEGCERRNY3JUSAHOAHOHPMZ75V5J6MQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:32:25.059574940 (40m40.840508328s) [noise=true , address=77.90.31.18:4822 , piece_id=BU7KV4R7HN5UTIQM5ZB5FSK53PWBDIWMZHWSS5YJFG347OGSTHZA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:16EACrmREmTxBUPQKwsABtESUoZKWzcxKCknE81xK1nrDYpZSg, address:77.90.31.18:4822): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:32:25.059680674 (40m40.840614041s) [seg_pos={0 0}, upload=615]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:32:25.088124602 (40m40.869057975s) [upload=615]: Done uploading segments... Jul 26 22:32:25.088153768 (40m40.869087129s) [upload=615]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 33.715086785s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.....................X......................................] duration: 456.542025ms FAIL (2025-07-26 22:33:52 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 456.562389ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [....................................................X.......] duration: 209.313242ms FAIL (2025-07-26 22:33:48 UTC) orbiter warning: uplink: too many requests Duration: 209.34397ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [......................................X........X.X.X...XX...] duration: 17.184905353s FAIL (2025-07-26 22:33:37 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 17.184943029s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [..X..X........XXX.XXX.X.XX..X..X......X.............X.......] duration: 447.304014ms FAIL (2025-07-26 22:33:01 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 447.345575ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..........XX..XXX..X..XX..X..XX.X....X....X........X........] duration: 434.62897ms FAIL (2025-07-26 22:32:58 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 434.656449ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [...X..X........XXX.XXX.X.XX..X..X......X.............X......] duration: 435.00656ms FAIL (2025-07-26 22:32:46 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 435.034962ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [......................X.....................................] duration: 546.386073ms FAIL (2025-07-26 22:32:44 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 546.415953ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.....................................................X......] duration: 207.121511ms FAIL (2025-07-26 22:32:37 UTC) orbiter warning: uplink: too many requests Duration: 207.155143ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX.] duration: 5.73014305s FAIL (2025-07-26 22:32:30 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0200 (39m38.953613576s) [upload=602]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 22:31:25.092817481 (39m40.873750866s) [upload=615]: Starting upload Jul 26 22:31:25.092917455 (39m40.873850841s) [upload=615]: Done starting upload Jul 26 22:31:25.093095667 (39m40.874029048s) [upload=615]: Uploading segments... Jul 26 22:31:25.093538635 (39m40.874472036s) [upload=615]: Got next segment. Inline: false Jul 26 22:31:25.093618508 (39m40.874551888s) [upload=615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:31:25.093624913 (39m40.874558297s) [upload=615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:31:25.131355162 (39m40.912288549s) [seg_pos={0 0}, upload=615]: Begin upload segment... Jul 26 22:31:25.131427766 (39m40.912361148s) [seg_pos={0 0}, upload=615]: Done begin upload segment. Jul 26 22:31:25.131463629 (39m40.912397017s) [noise=true , address=64.49.13.138:4482 , piece_id=DLWMFB534AQGVWQCALOCRHHIVQIORWED7G3YX3OPDLKK45ZYL7WQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.131879895 (39m40.91281328s) [noise=true , address=79.127.220.101:4072 , piece_id=XMWGQ3KT4HPYEXGLM4UVSGD24TRWC2XYIES2CF6SBI55CDACYAGA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.132543852 (39m40.91347724s) [noise=true , address=79.127.201.215:4351 , piece_id=TMNXYMBDACXBGB66DXK5FFEGCERRNY3JUSAHOAHOHPMZ75V5J6MQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.132579299 (39m40.913512688s) [noise=true , address=45.91.116.9:4269 , piece_id=MFXVLH73CJA24KMLKCVNJCZNKMEPPTT4OJRS467HSSAZ4EWEN3QA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133014796 (39m40.913948163s) [noise=true , address=79.127.205.246:4266 , piece_id=R6FB6SWR2WOUW6QTVYGU6VWEX3DPMFKFEXEKBNUM22KVYCIXZAAA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133072582 (39m40.914005971s) [noise=true , address=77.90.31.18:4822 , piece_id=BU7KV4R7HN5UTIQM5ZB5FSK53PWBDIWMZHWSS5YJFG347OGSTHZA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133510863 (39m40.914444253s) [noise=true , address=64.49.13.155:4918 , piece_id=62RZCYTG4WBNYDJBNJYI5ALYR32AEF4VLLL3UHKGT23AHU4CANVQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133529028 (39m40.914462418s) [noise=true , address=79.127.205.233:4656 , piece_id=M55U3Z7Q3IVZCBVMQOUNVLJOPPCQXDP3WJW4GS2BJB5JGK6YXEAQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133983589 (39m40.914917048s) [noise=true , address=79.127.205.234:4605 , piece_id=5SKQOIOWVTGSFTQ5UZBYFZUEMY6U5Q5HANHREPXKDUVUIGZJBQXQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134054538 (39m40.914987927s) [noise=true , address=79.127.205.235:4682 , piece_id=5T24674RXWOK7N5L6X75FEWYXUQGR5CEYZI4UUXT6OX624E5BXZQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134498895 (39m40.915432285s) [noise=true , address=77.90.31.21:4885 , piece_id=HYKGBHOAHXQDZBFQEURWPD7ANB2QPGY5DSDYCMFRLHJ4DSJAFQTQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134590489 (39m40.915523878s) [noise=true , address=79.127.219.48:4090 , piece_id=OCOSOC4AD3BXG7MR35TUXI4AXVPQXFHWYXC3QMLJZIBYM4UEUDVQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134966793 (39m40.915900183s) [noise=true , address=64.49.13.170:4721 , piece_id=5YM7KUCLHAYUDAJAIVORZ6QXDJOIH27NGKHOKBHYUW6ST6JRNLGA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135163729 (39m40.916097118s) [noise=true , address=156.146.43.225:4448 , piece_id=JCNIQ7BUJOVZN37PZHPCH7AKAKB5ONIIGB7BKXZRGVIPLMWFLWLQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135677391 (39m40.916610776s) [noise=true , address=79.127.219.38:4007 , piece_id=6ED5TDNA6C3CW3EOYUUWXSAZPAFKUAYOMJDNAYZXXL2HPAKCO2HA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135741108 (39m40.916674484s) [noise=true , address=45.91.116.11:4146 , piece_id=IFEFOFUQAR75XDLS5IVES5SDIIMGAT6R6ZVFH7XFQOUORMBOLRIA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.136284624 (39m40.917218003s) [noise=true , address=77.90.31.12:4608 , piece_id=RJ6ARFV3MTGZ34ELJPCKGVYHFNMKXJEVCT7JFF2CTMK5FMWXE5PQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.137524409 (39m40.918457794s) [noise=true , address=45.91.116.14:4222 , piece_id=BYVM6RJ4T3RQMJYVAROHAJCJX7MKGXJWN4O2B24ZR4GCN2PJ4EFQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.240969311 (39m41.021902693s) [upload=615]: Next returned nil segment Jul 26 22:31:25.240995734 (39m41.021929111s) [upload=615]: Waiting for error group managing segments... Jul 26 22:31:25.243994267 (39m41.024927648s) [noise=true , address=79.127.220.101:4072 , piece_id=XMWGQ3KT4HPYEXGLM4UVSGD24TRWC2XYIES2CF6SBI55CDACYAGA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12bQXjK4bv5583RV4BtjhC8WTDJgz255Ptvs6f5kqvJu4SA99mc, address:79.127.220.101:4072): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.245375223 (39m41.026308585s) [noise=true , address=79.127.219.38:4007 , piece_id=6ED5TDNA6C3CW3EOYUUWXSAZPAFKUAYOMJDNAYZXXL2HPAKCO2HA, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.291788207 (39m41.07272157s) [noise=true , address=77.90.31.12:4608 , piece_id=RJ6ARFV3MTGZ34ELJPCKGVYHFNMKXJEVCT7JFF2CTMK5FMWXE5PQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.388770962 (39m41.169704331s) [noise=true , address=45.91.116.11:4146 , piece_id=IFEFOFUQAR75XDLS5IVES5SDIIMGAT6R6ZVFH7XFQOUORMBOLRIA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.516260375 (39m41.297193757s) [noise=true , address=64.49.13.138:4482 , piece_id=DLWMFB534AQGVWQCALOCRHHIVQIORWED7G3YX3OPDLKK45ZYL7WQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.723538202 (39m41.504471568s) [noise=true , address=79.127.205.233:4656 , piece_id=M55U3Z7Q3IVZCBVMQOUNVLJOPPCQXDP3WJW4GS2BJB5JGK6YXEAQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.725133974 (39m41.506067342s) [noise=true , address=79.127.205.235:4682 , piece_id=5T24674RXWOK7N5L6X75FEWYXUQGR5CEYZI4UUXT6OX624E5BXZQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.748427597 (39m41.529360974s) [noise=true , address=79.127.205.246:4266 , piece_id=R6FB6SWR2WOUW6QTVYGU6VWEX3DPMFKFEXEKBNUM22KVYCIXZAAA, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.786472821 (39m41.567406198s) [noise=true , address=79.127.205.234:4605 , piece_id=5SKQOIOWVTGSFTQ5UZBYFZUEMY6U5Q5HANHREPXKDUVUIGZJBQXQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:26.151056459 (39m41.931989836s) [noise=true , address=45.91.116.9:4269 , piece_id=MFXVLH73CJA24KMLKCVNJCZNKMEPPTT4OJRS467HSSAZ4EWEN3QA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:26.431220045 (39m42.212153426s) [noise=true , address=45.91.116.14:4222 , piece_id=BYVM6RJ4T3RQMJYVAROHAJCJX7MKGXJWN4O2B24ZR4GCN2PJ4EFQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:26.952353227 (39m42.733286605s) [noise=true , address=64.49.13.170:4721 , piece_id=5YM7KUCLHAYUDAJAIVORZ6QXDJOIH27NGKHOKBHYUW6ST6JRNLGA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:27.155824153 (39m42.936757545s) [noise=true , address=156.146.43.225:4448 , piece_id=JCNIQ7BUJOVZN37PZHPCH7AKAKB5ONIIGB7BKXZRGVIPLMWFLWLQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:27.445015172 (39m43.225948552s) [noise=true , address=79.127.219.48:4090 , piece_id=OCOSOC4AD3BXG7MR35TUXI4AXVPQXFHWYXC3QMLJZIBYM4UEUDVQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:34.971697059 (39m50.752630428s) [noise=true , address=64.49.13.155:4918 , piece_id=62RZCYTG4WBNYDJBNJYI5ALYR32AEF4VLLL3UHKGT23AHU4CANVQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:42.955261339 (39m58.73619492s) [noise=true , address=77.90.31.21:4885 , piece_id=HYKGBHOAHXQDZBFQEURWPD7ANB2QPGY5DSDYCMFRLHJ4DSJAFQTQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:51.785580862 (40m7.566514236s) [noise=true , address=79.127.201.215:4351 , piece_id=TMNXYMBDACXBGB66DXK5FFEGCERRNY3JUSAHOAHOHPMZ75V5J6MQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:32:25.059574940 (40m40.840508328s) [noise=true , address=77.90.31.18:4822 , piece_id=BU7KV4R7HN5UTIQM5ZB5FSK53PWBDIWMZHWSS5YJFG347OGSTHZA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:16EACrmREmTxBUPQKwsABtESUoZKWzcxKCknE81xK1nrDYpZSg, address:77.90.31.18:4822): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:32:25.059680674 (40m40.840614041s) [seg_pos={0 0}, upload=615]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:32:25.088124602 (40m40.869057975s) [upload=615]: Done uploading segments... Jul 26 22:32:25.088153768 (40m40.869087129s) [upload=615]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 5.730167129s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.......................................X........X.X.X...XX..] duration: 4.834799903s FAIL (2025-07-26 22:32:27 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.834834902s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 38 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXXX] duration: 1m0.030676676s FAIL (2025-07-26 22:32:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== 0200 (39m38.953613576s) [upload=602]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Jul 26 22:31:25.092817481 (39m40.873750866s) [upload=615]: Starting upload Jul 26 22:31:25.092917455 (39m40.873850841s) [upload=615]: Done starting upload Jul 26 22:31:25.093095667 (39m40.874029048s) [upload=615]: Uploading segments... Jul 26 22:31:25.093538635 (39m40.874472036s) [upload=615]: Got next segment. Inline: false Jul 26 22:31:25.093618508 (39m40.874551888s) [upload=615]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:31:25.093624913 (39m40.874558297s) [upload=615]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:31:25.131355162 (39m40.912288549s) [seg_pos={0 0}, upload=615]: Begin upload segment... Jul 26 22:31:25.131427766 (39m40.912361148s) [seg_pos={0 0}, upload=615]: Done begin upload segment. Jul 26 22:31:25.131463629 (39m40.912397017s) [noise=true , address=64.49.13.138:4482 , piece_id=DLWMFB534AQGVWQCALOCRHHIVQIORWED7G3YX3OPDLKK45ZYL7WQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.131879895 (39m40.91281328s) [noise=true , address=79.127.220.101:4072 , piece_id=XMWGQ3KT4HPYEXGLM4UVSGD24TRWC2XYIES2CF6SBI55CDACYAGA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.132543852 (39m40.91347724s) [noise=true , address=79.127.201.215:4351 , piece_id=TMNXYMBDACXBGB66DXK5FFEGCERRNY3JUSAHOAHOHPMZ75V5J6MQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.132579299 (39m40.913512688s) [noise=true , address=45.91.116.9:4269 , piece_id=MFXVLH73CJA24KMLKCVNJCZNKMEPPTT4OJRS467HSSAZ4EWEN3QA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133014796 (39m40.913948163s) [noise=true , address=79.127.205.246:4266 , piece_id=R6FB6SWR2WOUW6QTVYGU6VWEX3DPMFKFEXEKBNUM22KVYCIXZAAA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133072582 (39m40.914005971s) [noise=true , address=77.90.31.18:4822 , piece_id=BU7KV4R7HN5UTIQM5ZB5FSK53PWBDIWMZHWSS5YJFG347OGSTHZA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133510863 (39m40.914444253s) [noise=true , address=64.49.13.155:4918 , piece_id=62RZCYTG4WBNYDJBNJYI5ALYR32AEF4VLLL3UHKGT23AHU4CANVQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133529028 (39m40.914462418s) [noise=true , address=79.127.205.233:4656 , piece_id=M55U3Z7Q3IVZCBVMQOUNVLJOPPCQXDP3WJW4GS2BJB5JGK6YXEAQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.133983589 (39m40.914917048s) [noise=true , address=79.127.205.234:4605 , piece_id=5SKQOIOWVTGSFTQ5UZBYFZUEMY6U5Q5HANHREPXKDUVUIGZJBQXQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134054538 (39m40.914987927s) [noise=true , address=79.127.205.235:4682 , piece_id=5T24674RXWOK7N5L6X75FEWYXUQGR5CEYZI4UUXT6OX624E5BXZQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134498895 (39m40.915432285s) [noise=true , address=77.90.31.21:4885 , piece_id=HYKGBHOAHXQDZBFQEURWPD7ANB2QPGY5DSDYCMFRLHJ4DSJAFQTQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134590489 (39m40.915523878s) [noise=true , address=79.127.219.48:4090 , piece_id=OCOSOC4AD3BXG7MR35TUXI4AXVPQXFHWYXC3QMLJZIBYM4UEUDVQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.134966793 (39m40.915900183s) [noise=true , address=64.49.13.170:4721 , piece_id=5YM7KUCLHAYUDAJAIVORZ6QXDJOIH27NGKHOKBHYUW6ST6JRNLGA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135163729 (39m40.916097118s) [noise=true , address=156.146.43.225:4448 , piece_id=JCNIQ7BUJOVZN37PZHPCH7AKAKB5ONIIGB7BKXZRGVIPLMWFLWLQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135677391 (39m40.916610776s) [noise=true , address=79.127.219.38:4007 , piece_id=6ED5TDNA6C3CW3EOYUUWXSAZPAFKUAYOMJDNAYZXXL2HPAKCO2HA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.135741108 (39m40.916674484s) [noise=true , address=45.91.116.11:4146 , piece_id=IFEFOFUQAR75XDLS5IVES5SDIIMGAT6R6ZVFH7XFQOUORMBOLRIA, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.136284624 (39m40.917218003s) [noise=true , address=77.90.31.12:4608 , piece_id=RJ6ARFV3MTGZ34ELJPCKGVYHFNMKXJEVCT7JFF2CTMK5FMWXE5PQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.137524409 (39m40.918457794s) [noise=true , address=45.91.116.14:4222 , piece_id=BYVM6RJ4T3RQMJYVAROHAJCJX7MKGXJWN4O2B24ZR4GCN2PJ4EFQ, seg_pos={0 0}, upload=615]: Uploading piece... Jul 26 22:31:25.240969311 (39m41.021902693s) [upload=615]: Next returned nil segment Jul 26 22:31:25.240995734 (39m41.021929111s) [upload=615]: Waiting for error group managing segments... Jul 26 22:31:25.243994267 (39m41.024927648s) [noise=true , address=79.127.220.101:4072 , piece_id=XMWGQ3KT4HPYEXGLM4UVSGD24TRWC2XYIES2CF6SBI55CDACYAGA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12bQXjK4bv5583RV4BtjhC8WTDJgz255Ptvs6f5kqvJu4SA99mc, address:79.127.220.101:4072): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.245375223 (39m41.026308585s) [noise=true , address=79.127.219.38:4007 , piece_id=6ED5TDNA6C3CW3EOYUUWXSAZPAFKUAYOMJDNAYZXXL2HPAKCO2HA, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.291788207 (39m41.07272157s) [noise=true , address=77.90.31.12:4608 , piece_id=RJ6ARFV3MTGZ34ELJPCKGVYHFNMKXJEVCT7JFF2CTMK5FMWXE5PQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.388770962 (39m41.169704331s) [noise=true , address=45.91.116.11:4146 , piece_id=IFEFOFUQAR75XDLS5IVES5SDIIMGAT6R6ZVFH7XFQOUORMBOLRIA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12VJf4TrDJ7So1W3XuGbfnbKMWC8ftHReLgqfWGqjpRWAbwCjrp, address:45.91.116.11:4146): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.516260375 (39m41.297193757s) [noise=true , address=64.49.13.138:4482 , piece_id=DLWMFB534AQGVWQCALOCRHHIVQIORWED7G3YX3OPDLKK45ZYL7WQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:12joW2m4Uc87FSH65Cep1UKivUB1HG1119CQj14eWEKPcwaPsD5, address:64.49.13.138:4482): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:25.723538202 (39m41.504471568s) [noise=true , address=79.127.205.233:4656 , piece_id=M55U3Z7Q3IVZCBVMQOUNVLJOPPCQXDP3WJW4GS2BJB5JGK6YXEAQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.725133974 (39m41.506067342s) [noise=true , address=79.127.205.235:4682 , piece_id=5T24674RXWOK7N5L6X75FEWYXUQGR5CEYZI4UUXT6OX624E5BXZQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.748427597 (39m41.529360974s) [noise=true , address=79.127.205.246:4266 , piece_id=R6FB6SWR2WOUW6QTVYGU6VWEX3DPMFKFEXEKBNUM22KVYCIXZAAA, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:25.786472821 (39m41.567406198s) [noise=true , address=79.127.205.234:4605 , piece_id=5SKQOIOWVTGSFTQ5UZBYFZUEMY6U5Q5HANHREPXKDUVUIGZJBQXQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:26.151056459 (39m41.931989836s) [noise=true , address=45.91.116.9:4269 , piece_id=MFXVLH73CJA24KMLKCVNJCZNKMEPPTT4OJRS467HSSAZ4EWEN3QA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1217KydURwsYXXhpRQHvgF5vJTTsjtedjW2AnwrZWA4cJ2VFy1Y, address:45.91.116.9:4269): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:26.431220045 (39m42.212153426s) [noise=true , address=45.91.116.14:4222 , piece_id=BYVM6RJ4T3RQMJYVAROHAJCJX7MKGXJWN4O2B24ZR4GCN2PJ4EFQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1HK7tc3kteH465RN26qhUuyPL8257whn1gMiuZDDP7gPYENQA9, address:45.91.116.14:4222): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:26.952353227 (39m42.733286605s) [noise=true , address=64.49.13.170:4721 , piece_id=5YM7KUCLHAYUDAJAIVORZ6QXDJOIH27NGKHOKBHYUW6ST6JRNLGA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:31:27.155824153 (39m42.936757545s) [noise=true , address=156.146.43.225:4448 , piece_id=JCNIQ7BUJOVZN37PZHPCH7AKAKB5ONIIGB7BKXZRGVIPLMWFLWLQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:27.445015172 (39m43.225948552s) [noise=true , address=79.127.219.48:4090 , piece_id=OCOSOC4AD3BXG7MR35TUXI4AXVPQXFHWYXC3QMLJZIBYM4UEUDVQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:34.971697059 (39m50.752630428s) [noise=true , address=64.49.13.155:4918 , piece_id=62RZCYTG4WBNYDJBNJYI5ALYR32AEF4VLLL3UHKGT23AHU4CANVQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:42.955261339 (39m58.73619492s) [noise=true , address=77.90.31.21:4885 , piece_id=HYKGBHOAHXQDZBFQEURWPD7ANB2QPGY5DSDYCMFRLHJ4DSJAFQTQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:31:51.785580862 (40m7.566514236s) [noise=true , address=79.127.201.215:4351 , piece_id=TMNXYMBDACXBGB66DXK5FFEGCERRNY3JUSAHOAHOHPMZ75V5J6MQ, seg_pos={0 0}, upload=615]: Done uploading piece. err: <nil> Jul 26 22:32:25.059574940 (40m40.840508328s) [noise=true , address=77.90.31.18:4822 , piece_id=BU7KV4R7HN5UTIQM5ZB5FSK53PWBDIWMZHWSS5YJFG347OGSTHZA, seg_pos={0 0}, upload=615]: Done uploading piece. err: ecclient: upload failed (node:16EACrmREmTxBUPQKwsABtESUoZKWzcxKCknE81xK1nrDYpZSg, address:77.90.31.18:4822): protocol: context deadline exceeded; context deadline exceeded Jul 26 22:32:25.059680674 (40m40.840614041s) [seg_pos={0 0}, upload=615]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 22:32:25.088124602 (40m40.869057975s) [upload=615]: Done uploading segments... Jul 26 22:32:25.088153768 (40m40.869087129s) [upload=615]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration: 1m0.03072904s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [......................................................X.....] duration: 247.309138ms FAIL (2025-07-26 22:32:05 UTC) orbiter warning: uplink: too many requests Duration: 247.346203ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [...........XX..XXX..X..XX..X..XX.X....X....X........X.......] duration: 641.280149ms FAIL (2025-07-26 22:31:59 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 641.315695ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [....X..X........XXX.XXX.X.XX..X..X......X.............X.....] duration: 541.603125ms FAIL (2025-07-26 22:31:54 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 541.63824ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.......................X....................................] duration: 587.336207ms FAIL (2025-07-26 22:31:48 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 587.418487ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 37 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXXX] duration: 1m0.029674709s FAIL (2025-07-26 22:31:23 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 13) ======LOGS====== iting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:30:08.761600623 (38m24.542534006s) [upload=582]: Done uploading segments... Jul 26 22:30:08.761635235 (38m24.542568618s) [upload=582]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Jul 26 22:30:23.175715785 (38m38.956649175s) [upload=602]: Starting upload Jul 26 22:30:23.176643641 (38m38.957577037s) [upload=602]: Done starting upload Jul 26 22:30:23.176858032 (38m38.957791403s) [upload=602]: Uploading segments... Jul 26 22:30:23.177442532 (38m38.958375909s) [upload=602]: Got next segment. Inline: false Jul 26 22:30:23.177529730 (38m38.958463096s) [upload=602]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:30:23.177539149 (38m38.958472517s) [upload=602]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:30:23.220426626 (38m39.001360014s) [seg_pos={0 0}, upload=602]: Begin upload segment... Jul 26 22:30:23.220496207 (38m39.001429595s) [seg_pos={0 0}, upload=602]: Done begin upload segment. Jul 26 22:30:23.220541622 (38m39.001475014s) [noise=true , address=79.127.201.215:4360 , piece_id=LDXCX2G3H2CUYF42CODPHX5A3DBFEVEGHR7NTBVUK2T6MIZAEM3Q, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.221481322 (38m39.002414711s) [noise=true , address=77.90.31.17:4796 , piece_id=LNOIFXAW622A3JSV3NXD3NFRSGYMPZ4ZYR62P3P6CJND4BJMEEIA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.222101054 (38m39.003034443s) [noise=true , address=79.127.205.234:4613 , piece_id=JRJH33VQB7YJTIGMXDKUZ2Z6JWD7S5WIUNJDJJFURPVA3W7IMZYA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.222767744 (38m39.003701132s) [noise=true , address=77.90.31.11:4580 , piece_id=UXKISZP65MHRIMHI3VILH7OAD3HJUQMEQ25EZCU6NZDYJF75QS7A, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.223371716 (38m39.004305103s) [noise=true , address=45.91.116.8:4065 , piece_id=ZWC5LB4UE3MN4FLC67IJUPR2OQCPM3YVFZTBL4IR2EZYRX363VFA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.223841406 (38m39.00477479s) [noise=true , address=45.91.116.14:4206 , piece_id=ZHFDNVYDBGHR4TAV4NYDQG7DCJ5YS6O7P6EWM6O5KH2RLJVE3P4A, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.224429753 (38m39.005363139s) [noise=true , address=77.90.31.21:4909 , piece_id=7BDNPKYYTBHFT34IUIXXKN7DM77M54PFKP7CUGWXTU3MMQUAMI3Q, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.224883320 (38m39.005816708s) [noise=true , address=45.91.116.5:4310 , piece_id=BX6SKOJY27LBLEIS3OKXUG2ISJBKBR2TEMBUBA6AQYWM3SGUHYVQ, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.225319227 (38m39.006252616s) [noise=true , address=64.49.13.169:4716 , piece_id=DISVDESQ35BEY3YDILQLFPKBI6KPNZTBKZSC4N2BFOHB6UATZM7Q, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.226710795 (38m39.007644183s) [noise=true , address=79.127.205.233:4652 , piece_id=PJUN5Q7AAGAIYBVFL2XVK7HOHN2VSUYYFMJFA2KAYUVKBR3GH2SQ, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.227512914 (38m39.00844641s) [noise=true , address=79.127.219.51:4201 , piece_id=7UOX3EMEC2TJKOD5TH5OQK7IGACKSDKS3427NS6LIACG2P3DGUDQ, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.227953319 (38m39.008886708s) [noise=true , address=64.49.13.170:4721 , piece_id=HJ233QGSCIENIKGWK6OC4FO6IHIICJY26LBG7ABUWQYHNDRL3JPQ, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.228555621 (38m39.00948901s) [noise=true , address=45.91.116.12:4229 , piece_id=HHVTYQRHUQHYJPZX3D3LH54QD545CE7VKALR2UL37IIVKIOLIPPA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.229094745 (38m39.010028132s) [noise=true , address=77.90.31.13:4651 , piece_id=5U273HIPJOE2ZPGFT6V5QMIVYMQGZ4AA4HMGLMVFEDVT7SXRGBQA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.229653467 (38m39.010586846s) [noise=true , address=79.127.205.245:4243 , piece_id=W6B3YDPNH2KGKQ3Q5UNMRV6JHFQZIPFVXPR6EXB6PIDUUVTMEOZA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.230260750 (38m39.011194139s) [noise=true , address=64.49.13.138:4483 , piece_id=X2TXDPN3IWXTA5WLA2QKPRCPW2S6SIQBWJ5E7I4DCEZA4C52YEPQ, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.230834024 (38m39.011767412s) [noise=true , address=77.90.31.15:4714 , piece_id=RCQ7JYIFAZ24CO7T5XFZWP3MK3QOFB57TX5PNQLKHIUDNGDU6MGQ, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.231466514 (38m39.012399898s) [noise=true , address=79.127.205.235:4682 , piece_id=NIUEICIFB67KGMLQVXEMSMPM32UKW622IPGF4F6RAW54IXIZMDMA, seg_pos={0 0}, upload=602]: Uploading piece... Jul 26 22:30:23.358902366 (38m39.139835729s) [upload=602]: Next returned nil segment Jul 26 22:30:23.358921206 (38m39.139854582s) [upload=602]: Waiting for error group managing segments... Jul 26 22:30:23.396711622 (38m39.177644997s) [noise=true , address=77.90.31.17:4796 , piece_id=LNOIFXAW622A3JSV3NXD3NFRSGYMPZ4ZYR62P3P6CJND4BJMEEIA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:23.429135315 (38m39.210068678s) [noise=true , address=77.90.31.13:4651 , piece_id=5U273HIPJOE2ZPGFT6V5QMIVYMQGZ4AA4HMGLMVFEDVT7SXRGBQA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:23.492792568 (38m39.273725937s) [noise=true , address=77.90.31.11:4580 , piece_id=UXKISZP65MHRIMHI3VILH7OAD3HJUQMEQ25EZCU6NZDYJF75QS7A, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:23.786085237 (38m39.567018615s) [noise=true , address=64.49.13.170:4721 , piece_id=HJ233QGSCIENIKGWK6OC4FO6IHIICJY26LBG7ABUWQYHNDRL3JPQ, seg_pos={0 0}, upload=602]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:23.833627956 (38m39.614561334s) [noise=true , address=79.127.205.233:4652 , piece_id=PJUN5Q7AAGAIYBVFL2XVK7HOHN2VSUYYFMJFA2KAYUVKBR3GH2SQ, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:23.850607137 (38m39.631540513s) [noise=true , address=79.127.205.245:4243 , piece_id=W6B3YDPNH2KGKQ3Q5UNMRV6JHFQZIPFVXPR6EXB6PIDUUVTMEOZA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:23.963675075 (38m39.744608451s) [noise=true , address=45.91.116.8:4065 , piece_id=ZWC5LB4UE3MN4FLC67IJUPR2OQCPM3YVFZTBL4IR2EZYRX363VFA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:24.037010439 (38m39.817943809s) [noise=true , address=79.127.205.234:4613 , piece_id=JRJH33VQB7YJTIGMXDKUZ2Z6JWD7S5WIUNJDJJFURPVA3W7IMZYA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:24.192023890 (38m39.972957259s) [noise=true , address=79.127.205.235:4682 , piece_id=NIUEICIFB67KGMLQVXEMSMPM32UKW622IPGF4F6RAW54IXIZMDMA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:24.994113890 (38m40.775047251s) [noise=true , address=79.127.219.51:4201 , piece_id=7UOX3EMEC2TJKOD5TH5OQK7IGACKSDKS3427NS6LIACG2P3DGUDQ, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:26.699149799 (38m42.480083167s) [noise=true , address=64.49.13.169:4716 , piece_id=DISVDESQ35BEY3YDILQLFPKBI6KPNZTBKZSC4N2BFOHB6UATZM7Q, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:28.523850606 (38m44.304783973s) [noise=true , address=45.91.116.14:4206 , piece_id=ZHFDNVYDBGHR4TAV4NYDQG7DCJ5YS6O7P6EWM6O5KH2RLJVE3P4A, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:28.684871960 (38m44.465805324s) [noise=true , address=45.91.116.12:4229 , piece_id=HHVTYQRHUQHYJPZX3D3LH54QD545CE7VKALR2UL37IIVKIOLIPPA, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:30:31.099634536 (38m46.880567918s) [noise=true , address=45.91.116.5:4310 , piece_id=BX6SKOJY27LBLEIS3OKXUG2ISJBKBR2TEMBUBA6AQYWM3SGUHYVQ, seg_pos={0 0}, upload=602]: Done uploading piece. err: ecclient: upload failed (node:127RMUc62xRq9RU5z3XfkZsq5sehmv3kP8o53tr9BxVcPE3GQCY, address:45.91.116.5:4310): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:31.341563839 (38m47.122497218s) [noise=true , address=64.49.13.138:4483 , piece_id=X2TXDPN3IWXTA5WLA2QKPRCPW2S6SIQBWJ5E7I4DCEZA4C52YEPQ, seg_pos={0 0}, upload=602]: Done uploading piece. err: ecclient: upload failed (node:12Bq23DceaRvomaGDHoZr3sYzSHHJEQr2wE4LzSEYk9rUm1DhLa, address:64.49.13.138:4483): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:54.205996972 (39m9.986930349s) [noise=true , address=79.127.201.215:4360 , piece_id=LDXCX2G3H2CUYF42CODPHX5A3DBFEVEGHR7NTBVUK2T6MIZAEM3Q, seg_pos={0 0}, upload=602]: Done uploading piece. err: <nil> Jul 26 22:31:23.146066980 (39m38.927000363s) [noise=true , address=77.90.31.15:4714 , piece_id=RCQ7JYIFAZ24CO7T5XFZWP3MK3QOFB57TX5PNQLKHIUDNGDU6MGQ, seg_pos={0 0}, upload=602]: Done uploading piece. err: ecclient: upload failed (node:12DTMJVUhaPKvEnvujCAk9byGb88QsevLbtCBB9R1mxS4DxShiP, address:77.90.31.15:4714): protocol: rpc: dial tcp 77.90.31.15:4714: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:31:23.146140712 (39m38.927074091s) [noise=true , address=77.90.31.21:4909 , piece_id=7BDNPKYYTBHFT34IUIXXKN7DM77M54PFKP7CUGWXTU3MMQUAMI3Q, seg_pos={0 0}, upload=602]: Done uploading piece. err: ecclient: upload failed (node:121RT7tP6czMFAiZ3DjzSSsYgPt3xZXVbXVpkcYxaZYscNiMjL6, address:77.90.31.21:4909): protocol: write tcp 159.223.143.39:45828->77.90.31.21:4909: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:31:23.146194450 (39m38.927127835s) [seg_pos={0 0}, upload=602]: Done waiting for segment. successful: 13 optimal: 14 errs: <nil> Jul 26 22:31:23.172654781 (39m38.953588151s) [upload=602]: Done uploading segments... Jul 26 22:31:23.172680200 (39m38.953613576s) [upload=602]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 13) Duration: 1m0.029725569s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [........................................X........X.X.X...XX.] duration: 4.987162153s FAIL (2025-07-26 22:31:20 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.987183675s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 6 out of 60 is failed (threshold=3) [.........................................X........X.X.X...XX] duration: 1m0.00063588s FAIL (2025-07-26 22:31:15 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.000652214s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [............XX..XXX..X..XX..X..XX.X....X....X........X......] duration: 1.114872713s FAIL (2025-07-26 22:31:04 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 1.114906933s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.......................................................X....] duration: 231.383667ms FAIL (2025-07-26 22:31:00 UTC) orbiter warning: uplink: too many requests Duration: 231.420682ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....X..X........XXX.XXX.X.XX..X..X......X.............X....] duration: 452.889304ms FAIL (2025-07-26 22:30:51 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 452.954895ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [........................................................X...] duration: 195.576028ms FAIL (2025-07-26 22:30:37 UTC) orbiter warning: uplink: too many requests Duration: 195.607825ms core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [........................X...................................] duration: 1.84352625s FAIL (2025-07-26 22:30:35 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.843580658s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [......X..X........XXX.XXX.X.XX..X..X......X.............X...] duration: 450.840526ms FAIL (2025-07-26 22:30:31 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 450.882768ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.............XX..XXX..X..XX..X..XX.X....X....X........X.....] duration: 562.993517ms FAIL (2025-07-26 22:30:20 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 563.175298ms core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [.......X..X........XXX.XXX.X.XX..X..X......X.............X..] duration: 515.983574ms FAIL (2025-07-26 22:30:16 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 516.038922ms core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 5 out of 60 is failed (threshold=3) [..........................................X........X.X.X...X] duration: 1m0.001177122s FAIL (2025-07-26 22:30:14 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1m0.001276067s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 36 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXXX] duration: 1m0.0392517s FAIL (2025-07-26 22:30:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 7) ======LOGS====== DRNV6QVZDYWTCCCEC7CNHCY3Z7YVMWEK3K2M4WSA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.872330524 (37m24.653263907s) [noise=true , address=64.49.13.170:4721 , piece_id=U4VAMA23OUVSBMM2ZWUH3RBRWV4R6IUAVOTMG57S7LKA4CLLDFYA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.872852006 (37m24.653785395s) [noise=true , address=104.36.82.41:4779 , piece_id=TWCQUEWFOQJP2HASYDMNVBO3A4Q7TUKCEPX346KFH3U6OHWYRJSA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.872977116 (37m24.653910502s) [noise=true , address=79.127.205.235:4692 , piece_id=GKFKK4SFO6KDDVXQMKRI6VUVKBWIGWRLLMRLZDYTKIIQJ3KF3KYQ, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.873348563 (37m24.654281945s) [noise=true , address=45.91.116.13:4175 , piece_id=KZQ25XSWEXYLXML66QVZLYYVO54FEYAZYCVFZNX27N3FHOFXIW4A, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.873460795 (37m24.654394183s) [noise=true , address=104.36.82.42:4813 , piece_id=4LHK3DNMMJLS4WEUTJ5X6BDLXXUAM2VLWZFKXQMASJYLKY7TZCWQ, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.873814983 (37m24.654748371s) [noise=true , address=104.36.82.45:4928 , piece_id=4VLYUSNGBNO2SMUSC2VYAHYCDWEU44F4FDWNUYMW6VDXKDHZ3UEQ, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.873904265 (37m24.654837652s) [noise=true , address=79.127.201.215:4354 , piece_id=SJKK5A7R4X5B4UBQDIRIKB65X2EFX7UAWGXWSFTVKZOI4NCSJUBA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.874272710 (37m24.655206097s) [noise=true , address=79.127.219.47:4048 , piece_id=SUE6QBJAZUSUGVDDBQC35ZU66K57MHRHZIIYESFME5SXLMUH7G4A, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.874350404 (37m24.655283792s) [noise=true , address=104.36.82.52:4088 , piece_id=4HYQBC7HQQXUDYARJAQYVZI34VLXFO7MUNI52YN2DFI4BHT6IGEQ, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.874729933 (37m24.655663317s) [noise=true , address=79.127.205.233:4652 , piece_id=BBD5SHWY6D2LOZOCSG734PYP3BC32EKRVG4WRKVG65L5FIKVBVBQ, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.874798536 (37m24.65573192s) [noise=true , address=104.36.82.11:4143 , piece_id=K4EPYJLKKIRGS565OPQO5CSP7W6DLSQKPG3O2JLIAMKBLDMKCOAA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.875192974 (37m24.656126364s) [noise=true , address=104.36.82.12:4160 , piece_id=K6XAPHAZWQK4EZE4AZFOI5LUWRO4PDSCCRLTUQM3BDV4RZ2IFRXA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.875280310 (37m24.656213703s) [noise=true , address=104.36.82.36:4651 , piece_id=NSLFJ3RIELJLG7TJXBB7X4NHZDCJ2J33ODTATJRNG3B4OPYAY62Q, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.875649341 (37m24.656582728s) [noise=true , address=45.91.116.14:4206 , piece_id=WKRPJDJC4GYUAX6GCV27HZABRW6CG2XPFFFMLL6ZDMCOVN2SRQIA, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.875758061 (37m24.656691447s) [noise=true , address=104.36.82.40:4942 , piece_id=KRPCZBHIU5ZBUV2PQZBRQ3ZJXSZZIPSNFRRLN3L65SYWDG4X5U4Q, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:08.876344348 (37m24.657277726s) [noise=true , address=79.127.205.245:4220 , piece_id=UW3UIJ6MPYAGZXSI7MOCF2ODRBD52VKSDG7GPKFYFZ7VH4KYTO2A, seg_pos={0 0}, upload=582]: Uploading piece... Jul 26 22:29:09.139975922 (37m24.920909285s) [upload=582]: Next returned nil segment Jul 26 22:29:09.140138981 (37m24.921072343s) [upload=582]: Waiting for error group managing segments... Jul 26 22:29:09.474617798 (37m25.255551165s) [noise=true , address=79.127.205.233:4652 , piece_id=BBD5SHWY6D2LOZOCSG734PYP3BC32EKRVG4WRKVG65L5FIKVBVBQ, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:29:09.476801572 (37m25.25773494s) [noise=true , address=79.127.205.235:4692 , piece_id=GKFKK4SFO6KDDVXQMKRI6VUVKBWIGWRLLMRLZDYTKIIQJ3KF3KYQ, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:29:09.561523995 (37m25.342457365s) [noise=true , address=79.127.205.245:4220 , piece_id=UW3UIJ6MPYAGZXSI7MOCF2ODRBD52VKSDG7GPKFYFZ7VH4KYTO2A, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:29:11.507316685 (37m27.288250074s) [noise=true , address=79.127.219.47:4048 , piece_id=SUE6QBJAZUSUGVDDBQC35ZU66K57MHRHZIIYESFME5SXLMUH7G4A, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:29:11.941822852 (37m27.722756237s) [noise=true , address=64.49.13.170:4721 , piece_id=U4VAMA23OUVSBMM2ZWUH3RBRWV4R6IUAVOTMG57S7LKA4CLLDFYA, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:1MYa3uS5XnUo4DEvPvXbjcKdCqddUUKbi7L6HV9q6VgyzehyD3, address:64.49.13.170:4721): protocol: storage node overloaded, request limit: 2000 Jul 26 22:29:12.834296579 (37m28.615229957s) [noise=true , address=79.127.201.215:4354 , piece_id=SJKK5A7R4X5B4UBQDIRIKB65X2EFX7UAWGXWSFTVKZOI4NCSJUBA, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:29:13.872012428 (37m29.652945807s) [noise=true , address=45.91.116.10:4100 , piece_id=VPHKMJGRLKOKDRNV6QVZDYWTCCCEC7CNHCY3Z7YVMWEK3K2M4WSA, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:12PQzzS3ZUKXCpiPHqTokRAC9MUuU8QQp7fJTDydmYehhs21YBn, address:45.91.116.10:4100): protocol: storage node overloaded, request limit: 2000 Jul 26 22:29:19.175006841 (37m34.955940211s) [noise=true , address=45.91.116.14:4206 , piece_id=WKRPJDJC4GYUAX6GCV27HZABRW6CG2XPFFFMLL6ZDMCOVN2SRQIA, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:29:21.249247070 (37m37.030180453s) [noise=true , address=45.91.116.13:4175 , piece_id=KZQ25XSWEXYLXML66QVZLYYVO54FEYAZYCVFZNX27N3FHOFXIW4A, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:124YLi7uuYNfk8TYrgdAkmrGnKmCfRC1MBmeJypA2hwHyWdp43U, address:45.91.116.13:4175): protocol: storage node overloaded, request limit: 2000 Jul 26 22:30:04.345974914 (38m20.126908287s) [noise=true , address=104.36.82.52:4088 , piece_id=4HYQBC7HQQXUDYARJAQYVZI34VLXFO7MUNI52YN2DFI4BHT6IGEQ, seg_pos={0 0}, upload=582]: Done uploading piece. err: <nil> Jul 26 22:30:08.725951783 (38m24.506885162s) [noise=true , address=104.36.82.40:4942 , piece_id=KRPCZBHIU5ZBUV2PQZBRQ3ZJXSZZIPSNFRRLN3L65SYWDG4X5U4Q, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:17o881pUMuG7K2dv83k7RkrJCrmM7kgUiWesgam4GEw21KVUn7, address:104.36.82.40:4942): protocol: rpc: dial tcp 104.36.82.40:4942: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726063777 (38m24.506997155s) [noise=true , address=104.36.82.41:4779 , piece_id=TWCQUEWFOQJP2HASYDMNVBO3A4Q7TUKCEPX346KFH3U6OHWYRJSA, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:1eRovJAp3AyUHP4qkuDvSSGYb9TG6U4tSNzniGJBeYLhkmiUVy, address:104.36.82.41:4779): protocol: rpc: dial tcp 104.36.82.41:4779: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726143819 (38m24.507077204s) [noise=true , address=104.36.82.11:4143 , piece_id=K4EPYJLKKIRGS565OPQO5CSP7W6DLSQKPG3O2JLIAMKBLDMKCOAA, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:12ipThd6PwZUpCXy3ML4qqMd9dmRLmxKjVNCNoT5wCg1szdWp8U, address:104.36.82.11:4143): protocol: rpc: dial tcp 104.36.82.11:4143: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726212930 (38m24.507146312s) [noise=true , address=104.36.82.12:4160 , piece_id=K6XAPHAZWQK4EZE4AZFOI5LUWRO4PDSCCRLTUQM3BDV4RZ2IFRXA, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:12dggZZRv5coYBNJEBvHNqWtMSqYkZoXYmitc9nBKJZftnsX17D, address:104.36.82.12:4160): protocol: rpc: dial tcp 104.36.82.12:4160: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726277487 (38m24.507210863s) [noise=true , address=104.36.82.56:4233 , piece_id=56UZRSJOWFMOUEGYU4CFLN4O5I4HHJ7JRPEULXON2XYAGROXDZPQ, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:1DmQeoQWTnFhQeJFseJoNYzFyo7vZjYKwDtQVJ6pomksWExsTb, address:104.36.82.56:4233): protocol: rpc: dial tcp 104.36.82.56:4233: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726339540 (38m24.507272924s) [noise=true , address=104.36.82.42:4813 , piece_id=4LHK3DNMMJLS4WEUTJ5X6BDLXXUAM2VLWZFKXQMASJYLKY7TZCWQ, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:1Egk1BCBmtjUZcwJKpoQjfrubA2czJKMEai5bzDqkPmXTRhaB6, address:104.36.82.42:4813): protocol: rpc: dial tcp 104.36.82.42:4813: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726402211 (38m24.507335589s) [noise=true , address=104.36.82.45:4928 , piece_id=4VLYUSNGBNO2SMUSC2VYAHYCDWEU44F4FDWNUYMW6VDXKDHZ3UEQ, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:12B6ymfqNvqiDsAbkF8H4ExxEuYaYb7gYQ45Y5GF4PGAAwmjbyd, address:104.36.82.45:4928): protocol: rpc: dial tcp 104.36.82.45:4928: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726467024 (38m24.50740041s) [noise=true , address=104.36.82.36:4651 , piece_id=NSLFJ3RIELJLG7TJXBB7X4NHZDCJ2J33ODTATJRNG3B4OPYAY62Q, seg_pos={0 0}, upload=582]: Done uploading piece. err: ecclient: upload failed (node:12tD1Zfi4ZxujF9pVdgW5k2wnejXsa7tf1u25FuucRXeneGP3dz, address:104.36.82.36:4651): protocol: write tcp 159.223.143.39:43616->104.36.82.36:4651: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:30:08.726489768 (38m24.507423159s) [seg_pos={0 0}, upload=582]: Done waiting for segment. successful: 7 optimal: 14 errs: <nil> Jul 26 22:30:08.761600623 (38m24.542534006s) [upload=582]: Done uploading segments... Jul 26 22:30:08.761635235 (38m24.542568618s) [upload=582]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 7) Duration: 1m0.039269014s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [........X..X........XXX.XXX.X.XX..X..X......X.............X.] duration: 2.001677008s FAIL (2025-07-26 22:29:48 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 2.001717878s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [.........................X..................................] duration: 515.273029ms FAIL (2025-07-26 22:29:47 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 515.306952ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [.........................................................X..] duration: 221.064152ms FAIL (2025-07-26 22:29:41 UTC) orbiter warning: uplink: too many requests Duration: 221.097603ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline) antiblip: 1 out of 60 is failed (threshold=3) [..........................................................X.] duration: 220.865789ms FAIL (2025-07-26 22:29:24 UTC) orbiter warning: uplink: too many requests Duration: 220.900138ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [..............XX..XXX..X..XX..X..XX.X....X....X........X....] duration: 456.197031ms FAIL (2025-07-26 22:29:19 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 12) ======LOGS====== g_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1YHj4FooV9PriUJCKWPDkd47nApVSUBCJwYXWADZodPLbuUZi3): protocol: expected piece hash; context canceled; EOF Jul 26 22:24:53.253735294 (33m9.034668658s) [noise=true , address=77.90.31.17:4770 , piece_id=OASIUXV542UMRKAC4FKNVDKITKQD6QICT6HPDDKH3RFAW34H6FTA, seg_pos={0 0}, upload=518]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:125ivanwRPxZhNW6ryony9FCETtFegxPRZg18B3AB2B9ch8865R): protocol: rpc: dial tcp 77.90.31.17:4770: operation was canceled; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:24:53.253754918 (33m9.034688282s) [seg_pos={0 0}, upload=518]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil> Jul 26 22:24:53.253841092 (33m9.034774467s) [upload=518]: Flush batch item: pb.BatchRequestItem_SegmentCommit Jul 26 22:24:53.253856986 (33m9.034790363s) [upload=518]: Flush batch item: pb.BatchRequestItem_ObjectCommit Jul 26 22:24:53.309939459 (33m9.090872826s) [upload=518]: Done uploading segments... Jul 26 22:24:53.309973445 (33m9.090906822s) [upload=518]: Upload finished. err: <nil> Jul 26 22:25:22.851455134 (33m38.63238851s) [upload=526]: Starting upload Jul 26 22:25:22.851579173 (33m38.632512559s) [upload=526]: Done starting upload Jul 26 22:25:22.851685400 (33m38.632618777s) [upload=526]: Uploading segments... Jul 26 22:25:22.851993415 (33m38.63292679s) [upload=526]: Got next segment. Inline: false Jul 26 22:25:22.852074964 (33m38.63300834s) [upload=526]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 22:25:22.852082321 (33m38.6330157s) [upload=526]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 22:25:22.880560155 (33m38.661493523s) [seg_pos={0 0}, upload=526]: Begin upload segment... Jul 26 22:25:22.880609032 (33m38.6615424s) [seg_pos={0 0}, upload=526]: Done begin upload segment. Jul 26 22:25:22.880635439 (33m38.661568823s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.880684964 (33m38.661618351s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881039210 (33m38.661972588s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881176690 (33m38.662110082s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881390312 (33m38.662323691s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881566215 (33m38.662499662s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881808536 (33m38.662741917s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.881904463 (33m38.662837842s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882224943 (33m38.663158329s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882263707 (33m38.663197088s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882574607 (33m38.663507992s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882675721 (33m38.663609103s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.882906628 (33m38.663840013s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883099195 (33m38.664032581s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883494748 (33m38.664428132s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.883908798 (33m38.664842183s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Uploading piece... Jul 26 22:25:22.981513752 (33m38.762447123s) [upload=526]: Next returned nil segment Jul 26 22:25:22.981532774 (33m38.762466162s) [upload=526]: Waiting for error group managing segments... Jul 26 22:25:22.985403565 (33m38.766336936s) [noise=true , address=45.91.116.12:4226 , piece_id=EWCA3HFLVYIBDMMDKCOZCANBOCNNPDFUZM3TPCUOLJYGG26KPN6Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986009557 (33m38.766942927s) [noise=true , address=45.91.116.4:4940 , piece_id=RW25SBSOW3KGOVM2D4XKRGT3MNH24RLFSMASKHX7IXHNPTAXFJWQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.986572245 (33m38.767505606s) [noise=true , address=45.91.116.11:4118 , piece_id=KFBGYAZS6DZXBFHJTSL7N62DDTCYKGC6RAK6V3KYLQML5ZB2S5OQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987250310 (33m38.768183691s) [noise=true , address=45.91.116.11:4119 , piece_id=SG5546QDUHBGHMMLDS743BZMRL6EVKCF76ULBBJTWHER4ZEQPSSA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.987786627 (33m38.768719995s) [noise=true , address=45.91.116.13:4154 , piece_id=M3LQAWYCBQQPCH2A3OZSSREGAEBBEZB55KY5MK25KIE32BQCZKPA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.988431106 (33m38.769364469s) [noise=true , address=45.91.116.5:4297 , piece_id=UTPJ53TH45BMUI6BEET4XV2L4MLYXNBOZR6Q72QFEHLMLVMPDAFA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989097420 (33m38.770030789s) [noise=true , address=77.90.31.14:4662 , piece_id=FA4EMAJY7R6332NTEYCYV4DUOPNTBS4QJJAQXOTAD5A7FDRJTH3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.989552082 (33m38.770485448s) [noise=true , address=45.91.116.5:4298 , piece_id=IKJG7CTT5M564KOWDG5KXWYZ2KQTBFLMK4PM6MV5LBLYRWLT6CHA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990074070 (33m38.771007433s) [noise=true , address=45.91.116.14:4191 , piece_id=UBT4JWTRRWPRZBQN74452KM2RN6FQQDCHKSPDHB4GDRCW2EITV3Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:22.990579158 (33m38.771512537s) [noise=true , address=45.91.116.10:4082 , piece_id=VXAQHPMRYBRC3AZXHWHD5WBCBRKRTFIQSQW5FVG4M2L77HRYDHDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:23.019257997 (33m38.800191367s) [noise=true , address=77.90.31.10:4519 , piece_id=27L7DYVH7D5ZR3VL3LGG6V7LMIUE5UMI6LLCZQM62UAMOISKPTDA, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:24.631743638 (33m40.412677022s) [noise=true , address=77.90.31.18:4807 , piece_id=BU2R3PB3R7RLOWQG5LTVWHVVGVL7QYEGMF2EDKQO5EGFSVG5MMFQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: <nil> Jul 26 22:25:52.823775591 (34m8.604708996s) [noise=true , address=45.91.116.8:4045 , piece_id=PAG323USYDRHSNGBKEHZB2GKJ6GL2XV4JBL2KSDKJSMQ4IFAU5IQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1VJaB7i7HKyemCadiPSoXCuTuBzvqa5MeUDQf793etkYLpxQc9, address:45.91.116.8:4045): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 22:25:52.823997629 (34m8.604931006s) [noise=true , address=77.90.31.12:4590 , piece_id=W336LBCRUCWRJSMW3ZEMKTS2SLS7VLQ52IPIHE7TREQN3S22QGQQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:1i9B5aDDfPkgveDvLh7ngsnCJb1Viw2cHYNMmgbjhG5BkWFCMw, address:77.90.31.12:4590): protocol: rpc: dial tcp 77.90.31.12:4590: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824056939 (34m8.604990316s) [noise=true , address=104.36.82.12:4149 , piece_id=N6MHKGYUPGWCM5XO24VPV37WYDCYXDT4UCOBDZR5AFT4BIWZZX5Q, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: rpc: dial tcp 104.36.82.12:4149: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824234902 (34m8.605168281s) [noise=true , address=77.90.31.21:4878 , piece_id=EQ3R3YWMQQPTG7X4SJJHY46IVCDKCHJELXHQJKG4ZHCI5QIXAEIQ, seg_pos={0 0}, upload=526]: Done uploading piece. err: ecclient: upload failed (node:12sFeLTeZ6ox1U2b5dDorbFq1mc5rPXdFjYgUCGFzRmRYv4oNpK, address:77.90.31.21:4878): protocol: write tcp 159.223.143.39:55166->77.90.31.21:4878: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:25:52.824261367 (34m8.60519475s) [seg_pos={0 0}, upload=526]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 22:25:52.850726652 (34m8.631660034s) [upload=526]: Done uploading segments... Jul 26 22:25:52.850774522 (34m8.631707899s) [upload=526]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Duration: 456.232604ms core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 1 out of 60 is failed (threshold=3) [..........................X.................................] duration: 611.778316ms FAIL (2025-07-26 22:29:19 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 611.807268ms edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 15 out of 60 is failed (threshold=3) [.........X..X........XXX.XXX.X.XX..X..X......X.............X] duration: 405.750901ms FAIL (2025-07-26 22:29:12 UTC) orbiter warning: uplink: invalid range 0:4194304 (size:4194304) Duration: 405.782476ms core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 35 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXX.XXXXXXXXX] duration: 1m0.03076395s FAIL (2025-07-26 22:29:08 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 8) ======LOGS====== eg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.781832705 (36m24.562766094s) [noise=true , address=79.127.201.215:4355 , piece_id=E2A5I7QPSE4BJ3GTRKVBMAZQ5WQLQOPDH4FWN5ZEIQU47AAZHM4A, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.782330540 (36m24.563264048s) [noise=true , address=79.127.205.245:4236 , piece_id=6BC2XEUIHGPXXN57ABQQU5HRACK2A6NQP7FNHI3XZ7LOB2WCXRFQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.782464497 (36m24.563397886s) [noise=true , address=79.127.205.246:4287 , piece_id=CCVRY4D4VQT5NPPTGPZHVSHFI3KQFC7I7L73VK56ZA35O3DL5QQQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.782878492 (36m24.563811878s) [noise=true , address=104.36.82.18:4393 , piece_id=CD2FO2WJDQPGBRV6XL2RVYFHQSD46LFFAAWBEQCTK4YT2GYINARA, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.783111638 (36m24.564045026s) [noise=true , address=104.36.82.40:4967 , piece_id=APXWWFDCX3R22UNOS6DU7TQZDOESBJU7IZAE7XK33VIKPDACJKGQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.783530994 (36m24.56446438s) [noise=true , address=79.127.219.38:4007 , piece_id=REDAVJSLA3XYRC6ADZ4JLUXE26GXTWBNMRPEWPYZXJ7VU4VVOH5A, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.783765643 (36m24.56469903s) [noise=true , address=104.36.82.44:4890 , piece_id=ZR2KTG6XPPKEUTKUPFC5C2LGEPNMYVBEMILEQTYXEJB3DY3CNUDQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.783946386 (36m24.564879778s) [noise=true , address=104.36.82.44:4897 , piece_id=FPU2DV6FSRXHCQZIFDWYWRZCGMMML5HLP22SET4M3XLVBO37E53Q, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.784412344 (36m24.565345728s) [noise=true , address=64.49.13.138:4483 , piece_id=T43HVJMVKJMD6WAA5QHO4URWNAWJNBP6FS7FPQYS6EKPGDC3KFVQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.784460953 (36m24.565394339s) [noise=true , address=104.36.82.56:4239 , piece_id=TTAN677QIV3CUW432OBTIX3DY7WLX7GNDD25OS6FMA5UYQMZGKGQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.784942001 (36m24.565875392s) [noise=true , address=77.90.31.18:4832 , piece_id=742YMK223EYD7JFFCB2TQBOQ6NG3OHT2E24MWO6XIJN2DIVTKU3A, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.786106520 (36m24.56703991s) [noise=true , address=104.36.82.7:4916 , piece_id=SRQG2L36L2I6OYFRMDUXVZNS3UMUKORVIASNTAWBPHYHKNA37JMA, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.786393751 (36m24.567327141s) [noise=true , address=104.36.82.10:4089 , piece_id=GFG6P4EYVFVWPMITY6OKTR5L2ICM6UZNBNPJUQL6TSED34LHNMXQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.786632451 (36m24.567565839s) [noise=true , address=79.127.219.48:4090 , piece_id=JPFMYBZTM2RW7HYKLUCTCM6UA6SUJDBCCA46J2WS3Q6BYNHMI7FQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.786826948 (36m24.567760337s) [noise=true , address=104.36.82.11:4125 , piece_id=IF5726HY35L7Q47QPCK7WMOJSBVFVJ7XQASG3BQZQMX4BJYEUYOQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.787078509 (36m24.568011888s) [noise=true , address=77.90.31.15:4726 , piece_id=JJ7Y6JR7BULYCKJVJPHAJXURC4AQDP3EOZQYIQNJSMHW3CB3GYMQ, seg_pos={0 0}, upload=568]: Uploading piece... Jul 26 22:28:08.869617862 (36m24.650551257s) [upload=568]: Next returned nil segment Jul 26 22:28:08.869639727 (36m24.650573114s) [upload=568]: Waiting for error group managing segments... Jul 26 22:28:08.903844258 (36m24.684777623s) [noise=true , address=77.90.31.18:4832 , piece_id=742YMK223EYD7JFFCB2TQBOQ6NG3OHT2E24MWO6XIJN2DIVTKU3A, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:08.997481088 (36m24.778414463s) [noise=true , address=79.127.219.38:4007 , piece_id=REDAVJSLA3XYRC6ADZ4JLUXE26GXTWBNMRPEWPYZXJ7VU4VVOH5A, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:09.125047128 (36m24.905980512s) [noise=true , address=79.127.219.48:4090 , piece_id=JPFMYBZTM2RW7HYKLUCTCM6UA6SUJDBCCA46J2WS3Q6BYNHMI7FQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:09.283982565 (36m25.064915935s) [noise=true , address=77.90.31.15:4726 , piece_id=JJ7Y6JR7BULYCKJVJPHAJXURC4AQDP3EOZQYIQNJSMHW3CB3GYMQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:09.389541409 (36m25.170474777s) [noise=true , address=79.127.205.245:4236 , piece_id=6BC2XEUIHGPXXN57ABQQU5HRACK2A6NQP7FNHI3XZ7LOB2WCXRFQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:09.477572185 (36m25.258505554s) [noise=true , address=79.127.205.246:4287 , piece_id=CCVRY4D4VQT5NPPTGPZHVSHFI3KQFC7I7L73VK56ZA35O3DL5QQQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:14.011496993 (36m29.792430379s) [noise=true , address=64.49.13.138:4483 , piece_id=T43HVJMVKJMD6WAA5QHO4URWNAWJNBP6FS7FPQYS6EKPGDC3KFVQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:28:22.943945444 (36m38.724878812s) [noise=true , address=79.127.201.215:4355 , piece_id=E2A5I7QPSE4BJ3GTRKVBMAZQ5WQLQOPDH4FWN5ZEIQU47AAZHM4A, seg_pos={0 0}, upload=568]: Done uploading piece. err: <nil> Jul 26 22:29:08.617901143 (37m24.39883452s) [noise=true , address=104.36.82.18:4393 , piece_id=CD2FO2WJDQPGBRV6XL2RVYFHQSD46LFFAAWBEQCTK4YT2GYINARA, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:12fDTUq74fpQVaFNsfxPmKwMpG9eUfFDcjPd5FutuqXiBp9R5EN, address:104.36.82.18:4393): protocol: rpc: dial tcp 104.36.82.18:4393: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.617992630 (37m24.398925992s) [noise=true , address=104.36.82.7:4916 , piece_id=SRQG2L36L2I6OYFRMDUXVZNS3UMUKORVIASNTAWBPHYHKNA37JMA, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:12fifEPABGWmVzxBHNzhjh3r4ue3RRSeNdydifpXruXtH4FELBK, address:104.36.82.7:4916): protocol: rpc: dial tcp 104.36.82.7:4916: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618231339 (37m24.399164717s) [noise=true , address=104.36.82.11:4125 , piece_id=IF5726HY35L7Q47QPCK7WMOJSBVFVJ7XQASG3BQZQMX4BJYEUYOQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:12vricofwx67FRaxAGLbmgT1TYyKkqfJbxdsyeEPL4R7kg8pKzT, address:104.36.82.11:4125): protocol: rpc: dial tcp 104.36.82.11:4125: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618318480 (37m24.399251863s) [noise=true , address=104.36.82.44:4897 , piece_id=FPU2DV6FSRXHCQZIFDWYWRZCGMMML5HLP22SET4M3XLVBO37E53Q, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:1xZEzok8AN2SUW6wups5Qug2TndRgrwkK2XMFvXfRyZcXFZRbU, address:104.36.82.44:4897): protocol: rpc: dial tcp 104.36.82.44:4897: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618394683 (37m24.39932806s) [noise=true , address=104.36.82.10:4089 , piece_id=GFG6P4EYVFVWPMITY6OKTR5L2ICM6UZNBNPJUQL6TSED34LHNMXQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:12bFi2opJ6FejAMwSL2qbqWoCk24QzSjT6xQYE8BytLgVJikZMT, address:104.36.82.10:4089): protocol: write tcp 159.223.143.39:34384->104.36.82.10:4089: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618462310 (37m24.399395687s) [noise=true , address=104.36.82.44:4890 , piece_id=ZR2KTG6XPPKEUTKUPFC5C2LGEPNMYVBEMILEQTYXEJB3DY3CNUDQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:1XYhMcqH63UiHYE4ofxDxjXz2HMvN1RfQjgxxxHcUst7P3aN7M, address:104.36.82.44:4890): protocol: rpc: dial tcp 104.36.82.44:4890: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618471039 (37m24.399404419s) [noise=true , address=104.36.82.52:4094 , piece_id=YFYPN3FRCIHUN22VGHWTK53JTQKDGPPVQVAWQGYORBLQWDKN7XQA, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:1QYpeobqZ5QtLcuNBh9fHhiKxnbEtpiAaagvgWyTZvGbgbFxH7, address:104.36.82.52:4094): protocol: rpc: dial tcp 104.36.82.52:4094: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618560947 (37m24.399494325s) [noise=true , address=104.36.82.56:4239 , piece_id=TTAN677QIV3CUW432OBTIX3DY7WLX7GNDD25OS6FMA5UYQMZGKGQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:126Feu3gWgog12YavTHZkkUKK8wq6RynV8zsWd3WUAUYjvQwBnc, address:104.36.82.56:4239): protocol: rpc: dial tcp 104.36.82.56:4239: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618599925 (37m24.399533288s) [noise=true , address=104.36.82.12:4152 , piece_id=J4Y7ZDRMQTXU6IVCFEC7U4JV23HQA47CYVQONKP62ASLXN6JYFVQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:122rsZGFcSywYKfw4DPU7duA9TGtj1iXmxS4ewGGGSXGRf7Q7mX, address:104.36.82.12:4152): protocol: write tcp 159.223.143.39:58020->104.36.82.12:4152: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618618191 (37m24.399551572s) [noise=true , address=104.36.82.40:4967 , piece_id=APXWWFDCX3R22UNOS6DU7TQZDOESBJU7IZAE7XK33VIKPDACJKGQ, seg_pos={0 0}, upload=568]: Done uploading piece. err: ecclient: upload failed (node:1Zz78xER2nL3XBQemnLxWTPUxtVySJv9yK2e7FfUcNeg5XK1r4, address:104.36.82.40:4967): protocol: write tcp 159.223.143.39:35046->104.36.82.40:4967: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 22:29:08.618677546 (37m24.399610934s) [seg_pos={0 0}, upload=568]: Done waiting for segment. successful: 8 optimal: 14 errs: <nil> Jul 26 22:29:08.644975149 (37m24.425908526s) [upload=568]: Done uploading segments... Jul 26 22:29:08.645009948 (37m24.425943394s) [upload=568]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 8) Duration: 1m0.030874016s core uplink us-select-1 us1-placement-20

Orbiter Check History

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.inline)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20)