Orbiter History

Orbiter: EU/US/AP/EDGE | Code | Config | Playbook | Escalation Procedure

Orbiter Check History

Showing 50 of 300164 reports (Page 5510 of 6004)

checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 46 out of 60 is failed (threshold=3) [........X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.032031617s FAIL (2025-07-26 23:43:27 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.032061499s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [............XXX.X....XX.X.X.X.......XXX...XX.....X.XXX......] duration: 9.545673361s FAIL (2025-07-26 23:43:27 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.545696426s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 42 out of 60 is failed (threshold=3) [.................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XXX] duration: 30.030371277s FAIL (2025-07-26 23:43:23 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== 3XARCEPTK4JYBWH2EUKMYTM62MULCUUFWOTC5M3BHWK7A, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.981083747 (1h51m9.762017143s) [noise=true , address=45.91.116.5:4297 , piece_id=25FDUAXC2C46CKAR4PJWS7CMVPOBWA6AQKMIZWJUBKUYF5RF67RQ, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.981492105 (1h51m9.76242548s) [noise=true , address=104.36.82.10:4077 , piece_id=OFBYIX2D5OOYKFB6A7RJGF2GW4N7HSHUQAMJRMNLEW7QFLQ5AO5Q, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.981516788 (1h51m9.762450154s) [noise=true , address=104.36.82.11:4115 , piece_id=GSJDZPDYFA7DMNTJ2TJFFUB4NCD6OCCZXH5YCICENLSB7RHPFD3Q, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.981849577 (1h51m9.762782978s) [noise=true , address=77.90.31.19:4841 , piece_id=LDYGHNFSQFAFN4O3I5BUFO64NAG6XYFGVK462CBYV24TJ52HU2EA, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.981870660 (1h51m9.762804039s) [noise=true , address=104.36.82.8:4972 , piece_id=DY6BGVJ5LKR6UUAJJVTOHU7YFHG6H6EZAKAKONZE5TP6YMAVW2ZQ, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.982170893 (1h51m9.763104268s) [noise=true , address=104.36.82.6:4873 , piece_id=5TKMXPN6YIVUNWM432J3SL7BQLVBJJ74UMQXC4WJHBIE65JS5PHA, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.982235052 (1h51m9.763168417s) [noise=true , address=104.36.82.13:4187 , piece_id=OWUCC5OVAOUEUXV2D43AOIJ6I7W2OPA6MJ6ENV4XPGGDITB3JCEQ, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.982584542 (1h51m9.763517917s) [noise=true , address=45.91.116.6:4974 , piece_id=4UFEKAEIBJNASQIVZM36MFXWRBU3SRGZJ34FQTHK2HKZBNJZUERQ, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.983051143 (1h51m9.763984526s) [noise=true , address=104.36.82.14:4222 , piece_id=JA7RJIRPTADTDJ73U6KLAFBRUETJR4ADEZLJZZJHWSQJ23F4MMEA, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.983636840 (1h51m9.764570218s) [noise=true , address=104.36.82.6:4875 , piece_id=SBWKH4VO7NPZTXUA24WEGKR37EFAVKAQIASWW63H4SBX2BZ6PKBA, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.984303355 (1h51m9.765236733s) [noise=true , address=104.36.82.12:4150 , piece_id=G25GIRL2W6DRKVSMO3NCQNQBNKB62BTCILHEBN4V47VCQRJ4F6FA, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.984627738 (1h51m9.765561119s) [noise=true , address=104.36.82.9:4006 , piece_id=A6I7YGR6CUHPDDSDTWCVQ5KL667SRBD7YMFVTZKYWVQST4S35XYA, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.984886950 (1h51m9.765820326s) [noise=true , address=104.36.82.7:4912 , piece_id=MVVOJM6FSRRPWCQO3JVRMZEOSKHGKMAYOGG3WUUAIRFPOX2LZRIQ, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.985246356 (1h51m9.766179732s) [noise=true , address=104.36.82.11:4113 , piece_id=JYVFDBOGWQOKH55W3YC5ZUD7PDBOGEEY4WUGAEQZ4OINYM6U2B6A, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:53.985640510 (1h51m9.766573886s) [noise=true , address=45.91.116.10:4081 , piece_id=6H4I445HNR3Y23Q4QD3WRHSPQ5775KSEMOC2WGVN3QSFO4XS4ADA, seg_pos={0 0}, upload=1703]: Uploading piece... Jul 26 23:42:54.046064822 (1h51m9.826998196s) [upload=1703]: Next returned nil segment Jul 26 23:42:54.046087221 (1h51m9.827020587s) [upload=1703]: Waiting for error group managing segments... Jul 26 23:42:54.049899328 (1h51m9.830832694s) [noise=true , address=45.91.116.10:4081 , piece_id=6H4I445HNR3Y23Q4QD3WRHSPQ5775KSEMOC2WGVN3QSFO4XS4ADA, seg_pos={0 0}, upload=1703]: Done uploading piece. err: <nil> Jul 26 23:42:54.051776328 (1h51m9.832709706s) [noise=true , address=45.91.116.6:4974 , piece_id=4UFEKAEIBJNASQIVZM36MFXWRBU3SRGZJ34FQTHK2HKZBNJZUERQ, seg_pos={0 0}, upload=1703]: Done uploading piece. err: <nil> Jul 26 23:42:54.063612686 (1h51m9.844546048s) [noise=true , address=77.90.31.18:4805 , piece_id=ZKZAFDU3XARCEPTK4JYBWH2EUKMYTM62MULCUUFWOTC5M3BHWK7A, seg_pos={0 0}, upload=1703]: Done uploading piece. err: <nil> Jul 26 23:42:54.416235746 (1h51m10.197169129s) [noise=true , address=45.91.116.5:4297 , piece_id=25FDUAXC2C46CKAR4PJWS7CMVPOBWA6AQKMIZWJUBKUYF5RF67RQ, seg_pos={0 0}, upload=1703]: Done uploading piece. err: <nil> Jul 26 23:42:56.087754955 (1h51m11.868688322s) [noise=true , address=77.90.31.19:4841 , piece_id=LDYGHNFSQFAFN4O3I5BUFO64NAG6XYFGVK462CBYV24TJ52HU2EA, seg_pos={0 0}, upload=1703]: Done uploading piece. err: <nil> Jul 26 23:43:23.916186840 (1h51m39.697120208s) [noise=true , address=104.36.82.11:4113 , piece_id=JYVFDBOGWQOKH55W3YC5ZUD7PDBOGEEY4WUGAEQZ4OINYM6U2B6A, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 159.223.143.39:45606->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916220069 (1h51m39.697153451s) [noise=true , address=104.36.82.9:4006 , piece_id=A6I7YGR6CUHPDDSDTWCVQ5KL667SRBD7YMFVTZKYWVQST4S35XYA, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: write tcp 159.223.143.39:56338->104.36.82.9:4006: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916255827 (1h51m39.697189201s) [noise=true , address=104.36.82.10:4077 , piece_id=OFBYIX2D5OOYKFB6A7RJGF2GW4N7HSHUQAMJRMNLEW7QFLQ5AO5Q, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916309256 (1h51m39.697242635s) [noise=true , address=104.36.82.13:4187 , piece_id=OWUCC5OVAOUEUXV2D43AOIJ6I7W2OPA6MJ6ENV4XPGGDITB3JCEQ, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: write tcp 159.223.143.39:49964->104.36.82.13:4187: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916333765 (1h51m39.697267132s) [noise=true , address=104.36.82.7:4912 , piece_id=MVVOJM6FSRRPWCQO3JVRMZEOSKHGKMAYOGG3WUUAIRFPOX2LZRIQ, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:58642->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916360143 (1h51m39.697293518s) [noise=true , address=104.36.82.11:4115 , piece_id=GSJDZPDYFA7DMNTJ2TJFFUB4NCD6OCCZXH5YCICENLSB7RHPFD3Q, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:39048->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916409312 (1h51m39.697342676s) [noise=true , address=104.36.82.8:4972 , piece_id=DY6BGVJ5LKR6UUAJJVTOHU7YFHG6H6EZAKAKONZE5TP6YMAVW2ZQ, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:12natVZCpE83KmTNWBmNV6FGRy62n2HtcodmHwyDwpoDaP8qVBX, address:104.36.82.8:4972): protocol: write tcp 159.223.143.39:41800->104.36.82.8:4972: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916436941 (1h51m39.697370316s) [noise=true , address=104.36.82.14:4222 , piece_id=JA7RJIRPTADTDJ73U6KLAFBRUETJR4ADEZLJZZJHWSQJ23F4MMEA, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:1ZLe9LXXbvrAbsYW3SKoqeihGsYhan3LgDBKjNrSspG6UnX4o1, address:104.36.82.14:4222): protocol: write tcp 159.223.143.39:50074->104.36.82.14:4222: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916455864 (1h51m39.697389229s) [noise=true , address=104.36.82.6:4873 , piece_id=5TKMXPN6YIVUNWM432J3SL7BQLVBJJ74UMQXC4WJHBIE65JS5PHA, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:53064->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916498734 (1h51m39.69743211s) [noise=true , address=104.36.82.6:4875 , piece_id=SBWKH4VO7NPZTXUA24WEGKR37EFAVKAQIASWW63H4SBX2BZ6PKBA, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: write tcp 159.223.143.39:39380->104.36.82.6:4875: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916552631 (1h51m39.697486011s) [noise=true , address=104.36.82.12:4150 , piece_id=G25GIRL2W6DRKVSMO3NCQNQBNKB62BTCILHEBN4V47VCQRJ4F6FA, seg_pos={0 0}, upload=1703]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:35634->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:43:23.916592747 (1h51m39.697526129s) [seg_pos={0 0}, upload=1703]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:43:23.944310881 (1h51m39.725244256s) [upload=1703]: Done uploading segments... Jul 26 23:43:23.944334367 (1h51m39.725267741s) [upload=1703]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.030394212s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.............XXX.X....XX.X.X.X.......XXX...XX.....X.XXX.....] duration: 1.887178471s FAIL (2025-07-26 23:42:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 1.887207055s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 41 out of 60 is failed (threshold=3) [..................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.XX] duration: 30.030356962s FAIL (2025-07-26 23:42:36 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 9) ======LOGS====== upload=1680]: Done uploading segments... Jul 26 23:42:06.214612402 (1h50m21.995545781s) [upload=1680]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Jul 26 23:42:06.305964647 (1h50m22.086898015s) [upload=1688]: Starting upload Jul 26 23:42:06.306063786 (1h50m22.086997165s) [upload=1688]: Done starting upload Jul 26 23:42:06.306143352 (1h50m22.087076719s) [upload=1688]: Uploading segments... Jul 26 23:42:06.306458206 (1h50m22.087391573s) [upload=1688]: Got next segment. Inline: false Jul 26 23:42:06.306520005 (1h50m22.087453388s) [upload=1688]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:42:06.306526960 (1h50m22.087460338s) [upload=1688]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:42:06.334335414 (1h50m22.115268803s) [seg_pos={0 0}, upload=1688]: Begin upload segment... Jul 26 23:42:06.334433210 (1h50m22.115366599s) [noise=true , address=104.36.82.3:4767 , piece_id=VQD4TZ7FGUJSZL3GM6BZMY5ZPR3VWLWSFUJSJT5SHGSYREXPTLNQ, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.334967292 (1h50m22.115900683s) [noise=true , address=104.36.82.10:4077 , piece_id=F4PNYXCLAWQKB3D4RR3MBWV6CMWVNAW6NAIZO4XFALQIIAFIWC2Q, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.335127934 (1h50m22.116061328s) [seg_pos={0 0}, upload=1688]: Done begin upload segment. Jul 26 23:42:06.335185809 (1h50m22.116119195s) [noise=true , address=77.90.31.16:4733 , piece_id=DJYNXBBGXFH4OZH3CRH4IP62TXMCLLGU5S5HZ3Q6U6LHHZSO6UZA, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.335752603 (1h50m22.116685985s) [noise=true , address=45.91.116.4:4938 , piece_id=L24WSVKYFX3CXOSUWIVIIDTK5XMPMK6GN4BNJYRESO6OFZSWBZ5Q, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.336310404 (1h50m22.117243793s) [noise=true , address=104.36.82.16:4295 , piece_id=PHQLTJRH2LCFMBPWSZ4MIFSISBCRMGW6QBUVHKXDRLI7SGYYR32A, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.336775211 (1h50m22.117708597s) [noise=true , address=77.90.31.13:4628 , piece_id=ANMDTRQ72JZXJZHRF4OSBLQR2XNHHH5X2SGSAMMN4A6A3G64Q3IA, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.337173662 (1h50m22.118107051s) [noise=true , address=45.91.116.13:4154 , piece_id=MD7PI4MKYZL5UWWTYGOM2KEFAVUUNX7JH7UH2WDCJX3MFSPRSA5Q, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.337563479 (1h50m22.118496868s) [noise=true , address=45.91.116.6:4973 , piece_id=6BLIJOO6U25IXC5XIWFL3PX5Q3JBXIIYRS6HHLMFERYQ6MJI3G7A, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.337974126 (1h50m22.118907516s) [noise=true , address=104.36.82.7:4909 , piece_id=RU6VERJ57PKID62X7ANIZQWSECLR7UYOCZNJMUDYDYSI4HXWYOBQ, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.338395884 (1h50m22.119329275s) [noise=true , address=104.36.82.6:4874 , piece_id=7BQ5N5PORURDW332WGJFGTLRSWCJZTMTJFCCI2BNA6YET5DGS4AA, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.338817490 (1h50m22.119750876s) [noise=true , address=45.91.116.14:4189 , piece_id=HNFVRFSSIJ6ZXTLRJSSY36GYA4FTR5575ROVESYPDZTWF5INAZ4A, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.339265321 (1h50m22.120198707s) [noise=true , address=104.36.82.15:4257 , piece_id=SQSCUDWJVGXQHAGPPCGQC5CDQFOKJUNKTECFRL26ENN4RAEI2LFQ, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.340228672 (1h50m22.121162061s) [noise=true , address=77.90.31.19:4841 , piece_id=WO32ZKFOQJM3TP7AXVMUJ6RLAK6M3Z2XTMQSHCFZ7F4443OLBVHA, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.340809116 (1h50m22.121742483s) [noise=true , address=45.91.116.5:4297 , piece_id=UGJNHDSFYMIZXY4AKNXQVC7PDVGUTUW5P4JSWVBGJ5FQSV2JM2MA, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.341233170 (1h50m22.122166538s) [noise=true , address=104.36.82.13:4186 , piece_id=JHWUTMTLE6WNEDVE7YMPQTJKRLPJNQZL7C4DMAWJLUW736F7R6XQ, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.341624887 (1h50m22.122558274s) [noise=true , address=45.91.116.8:4045 , piece_id=KUXVDG2OV7MLEXIMRH2YSFHXZVJWQJPX3AWS7F52S4M3CUU2HEBQ, seg_pos={0 0}, upload=1688]: Uploading piece... Jul 26 23:42:06.469607677 (1h50m22.250541057s) [upload=1688]: Next returned nil segment Jul 26 23:42:06.469640945 (1h50m22.250574324s) [upload=1688]: Waiting for error group managing segments... Jul 26 23:42:06.475366026 (1h50m22.256299392s) [noise=true , address=45.91.116.6:4973 , piece_id=6BLIJOO6U25IXC5XIWFL3PX5Q3JBXIIYRS6HHLMFERYQ6MJI3G7A, seg_pos={0 0}, upload=1688]: Done uploading piece. err: <nil> Jul 26 23:42:06.475737541 (1h50m22.25667091s) [noise=true , address=45.91.116.13:4154 , piece_id=MD7PI4MKYZL5UWWTYGOM2KEFAVUUNX7JH7UH2WDCJX3MFSPRSA5Q, seg_pos={0 0}, upload=1688]: Done uploading piece. err: <nil> Jul 26 23:42:06.476295326 (1h50m22.257228701s) [noise=true , address=45.91.116.8:4045 , piece_id=KUXVDG2OV7MLEXIMRH2YSFHXZVJWQJPX3AWS7F52S4M3CUU2HEBQ, seg_pos={0 0}, upload=1688]: Done uploading piece. err: <nil> Jul 26 23:42:06.476502798 (1h50m22.257436168s) [noise=true , address=45.91.116.4:4938 , piece_id=L24WSVKYFX3CXOSUWIVIIDTK5XMPMK6GN4BNJYRESO6OFZSWBZ5Q, seg_pos={0 0}, upload=1688]: Done uploading piece. err: <nil> Jul 26 23:42:06.513349488 (1h50m22.294282857s) [noise=true , address=45.91.116.5:4297 , piece_id=UGJNHDSFYMIZXY4AKNXQVC7PDVGUTUW5P4JSWVBGJ5FQSV2JM2MA, seg_pos={0 0}, upload=1688]: Done uploading piece. err: <nil> Jul 26 23:42:06.574425571 (1h50m22.355358943s) [noise=true , address=45.91.116.14:4189 , piece_id=HNFVRFSSIJ6ZXTLRJSSY36GYA4FTR5575ROVESYPDZTWF5INAZ4A, seg_pos={0 0}, upload=1688]: Done uploading piece. err: <nil> Jul 26 23:42:07.860144723 (1h50m23.641078102s) [noise=true , address=77.90.31.19:4841 , piece_id=WO32ZKFOQJM3TP7AXVMUJ6RLAK6M3Z2XTMQSHCFZ7F4443OLBVHA, seg_pos={0 0}, upload=1688]: Done uploading piece. err: <nil> Jul 26 23:42:07.926074677 (1h50m23.707008044s) [noise=true , address=77.90.31.13:4628 , piece_id=ANMDTRQ72JZXJZHRF4OSBLQR2XNHHH5X2SGSAMMN4A6A3G64Q3IA, seg_pos={0 0}, upload=1688]: Done uploading piece. err: <nil> Jul 26 23:42:08.680110898 (1h50m24.461044262s) [noise=true , address=77.90.31.16:4733 , piece_id=DJYNXBBGXFH4OZH3CRH4IP62TXMCLLGU5S5HZ3Q6U6LHHZSO6UZA, seg_pos={0 0}, upload=1688]: Done uploading piece. err: <nil> Jul 26 23:42:36.271778029 (1h50m52.05271141s) [noise=true , address=104.36.82.13:4186 , piece_id=JHWUTMTLE6WNEDVE7YMPQTJKRLPJNQZL7C4DMAWJLUW736F7R6XQ, seg_pos={0 0}, upload=1688]: Done uploading piece. err: ecclient: upload failed (node:16iUE9RyvrbkRAv7DZVHPsf9fk5ukduiYriLkcS43YNgxu66ht, address:104.36.82.13:4186): protocol: write tcp 159.223.143.39:52146->104.36.82.13:4186: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:36.271862783 (1h50m52.052796166s) [noise=true , address=104.36.82.7:4909 , piece_id=RU6VERJ57PKID62X7ANIZQWSECLR7UYOCZNJMUDYDYSI4HXWYOBQ, seg_pos={0 0}, upload=1688]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:36.271924435 (1h50m52.052857814s) [noise=true , address=104.36.82.10:4077 , piece_id=F4PNYXCLAWQKB3D4RR3MBWV6CMWVNAW6NAIZO4XFALQIIAFIWC2Q, seg_pos={0 0}, upload=1688]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: write tcp 159.223.143.39:53766->104.36.82.10:4077: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:36.271998960 (1h50m52.052932343s) [noise=true , address=104.36.82.6:4874 , piece_id=7BQ5N5PORURDW332WGJFGTLRSWCJZTMTJFCCI2BNA6YET5DGS4AA, seg_pos={0 0}, upload=1688]: Done uploading piece. err: ecclient: upload failed (node:12ay3ZqUvNcaX19JYT1hEaqydsxP6VoDjPavWXBNB34cestcm71, address:104.36.82.6:4874): protocol: write tcp 159.223.143.39:52030->104.36.82.6:4874: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:36.272011472 (1h50m52.052944845s) [noise=true , address=104.36.82.16:4295 , piece_id=PHQLTJRH2LCFMBPWSZ4MIFSISBCRMGW6QBUVHKXDRLI7SGYYR32A, seg_pos={0 0}, upload=1688]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: write tcp 159.223.143.39:53500->104.36.82.16:4295: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:36.272024315 (1h50m52.052957688s) [noise=true , address=104.36.82.3:4767 , piece_id=VQD4TZ7FGUJSZL3GM6BZMY5ZPR3VWLWSFUJSJT5SHGSYREXPTLNQ, seg_pos={0 0}, upload=1688]: Done uploading piece. err: ecclient: upload failed (node:1TiDHs6xp3atLDHHv1i2564ibyPGLKfFyHx3bgZWGRRN5cwUi5, address:104.36.82.3:4767): protocol: rpc: dial tcp 104.36.82.3:4767: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:36.272255843 (1h50m52.053189221s) [noise=true , address=104.36.82.15:4257 , piece_id=SQSCUDWJVGXQHAGPPCGQC5CDQFOKJUNKTECFRL26ENN4RAEI2LFQ, seg_pos={0 0}, upload=1688]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: write tcp 159.223.143.39:46470->104.36.82.15:4257: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:36.272285115 (1h50m52.053218506s) [seg_pos={0 0}, upload=1688]: Done waiting for segment. successful: 9 optimal: 13 errs: <nil> Jul 26 23:42:36.298684003 (1h50m52.079617373s) [upload=1688]: Done uploading segments... Jul 26 23:42:36.298733221 (1h50m52.07966661s) [upload=1688]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.030377305s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.....XX...X...X..XX.X...X..X.......X...X....XX..............] duration: 42.580725782s FAIL (2025-07-26 23:42:35 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 42.580749279s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 51 out of 60 is failed (threshold=3) [XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.033203436s FAIL (2025-07-26 23:42:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 40.038765653s) [upload=1676]: Got next segment. Inline: false Jul 26 23:41:24.257891756 (1h49m40.038825132s) [upload=1676]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:41:24.257905676 (1h49m40.038839039s) [upload=1676]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:41:24.368560933 (1h49m40.14949431s) [seg_pos={0 0}, upload=1676]: Begin upload segment... Jul 26 23:41:24.368634748 (1h49m40.149568125s) [seg_pos={0 0}, upload=1676]: Done begin upload segment. Jul 26 23:41:24.368661312 (1h49m40.149594696s) [noise=true , address=64.49.13.139:4478 , piece_id=SEALYRUU6WMN5DFB5HTFJQ4KGR4MOMGM4RNXEBEH4SRU55YXQA6Q, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.368671444 (1h49m40.149604823s) [noise=true , address=104.36.82.46:4502 , piece_id=GIYTZMLOJ3OUGDWGH3TCKTHBPTGSZTACX7EB3SNPHHG3TDWGNLOQ, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.368758256 (1h49m40.149691641s) [noise=true , address=45.91.116.9:4273 , piece_id=SQEXEHC5KJH6T6METISM65F7UM2ZMNGIRPSS5HDU2BV7UTAGA6AQ, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.369206778 (1h49m40.150140163s) [noise=true , address=79.127.205.234:4616 , piece_id=RQXLOBMU7RVBZ2IU3SGXNI65FNNHHISIER6X2ATLMFTTNHFAOULQ, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.369671630 (1h49m40.150605015s) [noise=true , address=64.49.13.137:4502 , piece_id=7FS5TCVFPX57CCC4G4HIDA2ANR5KZT7ZDBMUAVAYEAZ5RNPYE5JA, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.369996867 (1h49m40.150930246s) [noise=true , address=77.90.31.12:4608 , piece_id=L462NKLDIOXJS4APFQBLL3LD6XMOVMN7COOPRJPSRPYM622BT65Q, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.370114633 (1h49m40.151048018s) [noise=true , address=104.36.82.53:4129 , piece_id=7R77RGD2BNDAT3GWPOGJ3DUO74H7BPR7EF7HDSCLPQBSKNR5BBIQ, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.370583399 (1h49m40.151516783s) [noise=true , address=79.127.220.98:4422 , piece_id=R473XJFLXHJYFDAMFOLY3DGANAJPA2NGWQWQDGV3G6QIO6U2PQYA, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.371130862 (1h49m40.152064285s) [noise=true , address=79.127.205.246:4273 , piece_id=HSXMYGCUUELLUGLCJU4YMF2M5KDZ6AJ2IOCP2LTD7KSCC2ZUB4BQ, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.371661566 (1h49m40.152594951s) [noise=true , address=104.36.82.20:4471 , piece_id=334D667P6G4477HV737Y632Y45JC5DBFJ4UMJ2FPS4GMKOLGWDLQ, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.372506630 (1h49m40.153440015s) [noise=true , address=156.146.43.225:4448 , piece_id=Y7NN643J5DGFQBRXUYU6Z3I5STAK7EMKUZQOYRNKPLW425W4XSTA, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.373070034 (1h49m40.154003412s) [noise=true , address=79.127.201.215:4348 , piece_id=KWO7TJFJCNXQ7FJZNH6WNVUT4YTYINRQPBFJOQB26WLPXBRDUUSA, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.374911360 (1h49m40.155844737s) [noise=true , address=77.90.31.18:4809 , piece_id=ATDNAIDAY7P2S4V6L32DLTMKYV2NEO34Z7RT4VXEA5LDAE62JBUA, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.376159240 (1h49m40.15709263s) [noise=true , address=45.91.116.14:4222 , piece_id=KLJG4TGSOIZFZH5IRCZW6SPAES4CKP77UQFX54HZJWHIPQIP65JA, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.376631014 (1h49m40.157564399s) [noise=true , address=104.36.82.4:4819 , piece_id=EQMTLTI6TZ5RTOIQWT7TZAWOVLXEXXCTKZ7J6GXOXOCD3JWLTE6Q, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.377066117 (1h49m40.157999505s) [noise=true , address=79.127.205.233:4645 , piece_id=TDCWGLKTOEBVK3FSFEVSDOQTYOWD64SMMGGRLZV3TQA6UUODSTUA, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.377503897 (1h49m40.158437279s) [noise=true , address=104.36.82.55:4207 , piece_id=2OAPI5VWAYZOVWJBSLUAZW5YD3ABDJYVCNHHREWP3BWM375PGYCA, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.377954027 (1h49m40.158887411s) [noise=true , address=104.36.82.47:4535 , piece_id=CX42IE2RSAXADFH4MXY5NFMNX7WO5BELSSVZRVALINV37M37O3XA, seg_pos={0 0}, upload=1676]: Uploading piece... Jul 26 23:41:24.453937149 (1h49m40.234870521s) [upload=1676]: Next returned nil segment Jul 26 23:41:24.453959086 (1h49m40.234892465s) [upload=1676]: Waiting for error group managing segments... Jul 26 23:41:24.458611613 (1h49m40.23954499s) [noise=true , address=77.90.31.18:4809 , piece_id=ATDNAIDAY7P2S4V6L32DLTMKYV2NEO34Z7RT4VXEA5LDAE62JBUA, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:24.476428918 (1h49m40.257362296s) [noise=true , address=77.90.31.12:4608 , piece_id=L462NKLDIOXJS4APFQBLL3LD6XMOVMN7COOPRJPSRPYM622BT65Q, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:24.951783253 (1h49m40.732716618s) [noise=true , address=79.127.205.233:4645 , piece_id=TDCWGLKTOEBVK3FSFEVSDOQTYOWD64SMMGGRLZV3TQA6UUODSTUA, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:24.958568263 (1h49m40.739501634s) [noise=true , address=79.127.205.246:4273 , piece_id=HSXMYGCUUELLUGLCJU4YMF2M5KDZ6AJ2IOCP2LTD7KSCC2ZUB4BQ, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:24.976208837 (1h49m40.757142209s) [noise=true , address=79.127.205.234:4616 , piece_id=RQXLOBMU7RVBZ2IU3SGXNI65FNNHHISIER6X2ATLMFTTNHFAOULQ, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:25.045268499 (1h49m40.826201862s) [noise=true , address=64.49.13.139:4478 , piece_id=SEALYRUU6WMN5DFB5HTFJQ4KGR4MOMGM4RNXEBEH4SRU55YXQA6Q, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:25.079236000 (1h49m40.860169366s) [noise=true , address=64.49.13.137:4502 , piece_id=7FS5TCVFPX57CCC4G4HIDA2ANR5KZT7ZDBMUAVAYEAZ5RNPYE5JA, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:26.565677607 (1h49m42.346610973s) [noise=true , address=156.146.43.225:4448 , piece_id=Y7NN643J5DGFQBRXUYU6Z3I5STAK7EMKUZQOYRNKPLW425W4XSTA, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:28.689509351 (1h49m44.47044272s) [noise=true , address=79.127.220.98:4422 , piece_id=R473XJFLXHJYFDAMFOLY3DGANAJPA2NGWQWQDGV3G6QIO6U2PQYA, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:31.511898241 (1h49m47.292831605s) [noise=true , address=45.91.116.14:4222 , piece_id=KLJG4TGSOIZFZH5IRCZW6SPAES4CKP77UQFX54HZJWHIPQIP65JA, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:35.832008124 (1h49m51.612941487s) [noise=true , address=45.91.116.9:4273 , piece_id=SQEXEHC5KJH6T6METISM65F7UM2ZMNGIRPSS5HDU2BV7UTAGA6AQ, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:41:49.897102300 (1h50m5.678035672s) [noise=true , address=79.127.201.215:4348 , piece_id=KWO7TJFJCNXQ7FJZNH6WNVUT4YTYINRQPBFJOQB26WLPXBRDUUSA, seg_pos={0 0}, upload=1676]: Done uploading piece. err: <nil> Jul 26 23:42:24.227528079 (1h50m40.008461444s) [noise=true , address=104.36.82.47:4535 , piece_id=CX42IE2RSAXADFH4MXY5NFMNX7WO5BELSSVZRVALINV37M37O3XA, seg_pos={0 0}, upload=1676]: Done uploading piece. err: ecclient: upload failed (node:12KFwCcqBCTyeBBQYzg6SFG4YDzBm2H9s4D2JRvnC31VAGvpLhu, address:104.36.82.47:4535): protocol: rpc: dial tcp 104.36.82.47:4535: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:24.227587774 (1h50m40.008521138s) [noise=true , address=104.36.82.53:4129 , piece_id=7R77RGD2BNDAT3GWPOGJ3DUO74H7BPR7EF7HDSCLPQBSKNR5BBIQ, seg_pos={0 0}, upload=1676]: Done uploading piece. err: ecclient: upload failed (node:1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, address:104.36.82.53:4129): protocol: rpc: dial tcp 104.36.82.53:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:24.227712284 (1h50m40.00864566s) [noise=true , address=104.36.82.20:4471 , piece_id=334D667P6G4477HV737Y632Y45JC5DBFJ4UMJ2FPS4GMKOLGWDLQ, seg_pos={0 0}, upload=1676]: Done uploading piece. err: ecclient: upload failed (node:12DMRnzj9hpP9tRn5DUz9WcU3N4jEZ3CZ9saVTFa1tefjDNs72r, address:104.36.82.20:4471): protocol: rpc: dial tcp 104.36.82.20:4471: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:24.227760043 (1h50m40.008693409s) [noise=true , address=104.36.82.46:4502 , piece_id=GIYTZMLOJ3OUGDWGH3TCKTHBPTGSZTACX7EB3SNPHHG3TDWGNLOQ, seg_pos={0 0}, upload=1676]: Done uploading piece. err: ecclient: upload failed (node:12HP5xuRAGgTsL11UQh7VRgfuFzwNdEMXgc8zPVv4QknStyv6Ta, address:104.36.82.46:4502): protocol: rpc: dial tcp 104.36.82.46:4502: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:24.227779152 (1h50m40.008712527s) [noise=true , address=104.36.82.4:4819 , piece_id=EQMTLTI6TZ5RTOIQWT7TZAWOVLXEXXCTKZ7J6GXOXOCD3JWLTE6Q, seg_pos={0 0}, upload=1676]: Done uploading piece. err: ecclient: upload failed (node:12mh9u38YmD7NcPA5uB1KLaw8dToTUKywJsi1MhhSc17LXo15Gb, address:104.36.82.4:4819): protocol: rpc: dial tcp 104.36.82.4:4819: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:24.227821940 (1h50m40.008755317s) [noise=true , address=104.36.82.55:4207 , piece_id=2OAPI5VWAYZOVWJBSLUAZW5YD3ABDJYVCNHHREWP3BWM375PGYCA, seg_pos={0 0}, upload=1676]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:36586->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:24.227859111 (1h50m40.008792492s) [seg_pos={0 0}, upload=1676]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:42:24.258470670 (1h50m40.039404039s) [upload=1676]: Done uploading segments... Jul 26 23:42:24.258497044 (1h50m40.03943041s) [upload=1676]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.033227379s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 45 out of 60 is failed (threshold=3) [.........X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.025847775s FAIL (2025-07-26 23:42:20 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.025888515s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 40 out of 60 is failed (threshold=3) [...................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.X] duration: 30.030066823s FAIL (2025-07-26 23:42:06 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== TAGMQJGUC7HAHAJUI7I5ITNSEFAAUIK653A, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.255757677 (1h49m52.036691061s) [noise=true , address=104.36.82.14:4221 , piece_id=BAZUCKNRWY6FIMXJXC5SDMZ3NNMGQLKJYVG2ZNXWMXUIAXZD4UKQ, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.256412961 (1h49m52.03734635s) [noise=true , address=104.36.82.2:4731 , piece_id=FNTV3CVI67HG4BGPO4CGMY2XB5AFFCA5JN7OKYA5TX4D6TSEIKHA, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.256999306 (1h49m52.037932684s) [noise=true , address=104.36.82.12:4149 , piece_id=EPG5G6CKA3REHAH73ONGOIDNGFVYXRHBBWCEBB7WL5M66IBPC2KQ, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.257571391 (1h49m52.038504777s) [noise=true , address=45.91.116.6:4975 , piece_id=WXCTWLZWW23ZIDSORXNWNO7AHTRUWGEDUSY7X66BUHR6ZQSDJTBA, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.258077324 (1h49m52.03901071s) [noise=true , address=104.36.82.13:4185 , piece_id=YUNDQD3US3UDJ2YBTSHFQJN3WGSNJYE6BLHYYK5RJ653M27JUZOA, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.258637468 (1h49m52.039570855s) [noise=true , address=77.90.31.14:4661 , piece_id=RQ5BJWUNECNOOTVGJ257N2KHI5FTZ4UQ7RP7HTXLCYNLN64UXT2Q, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.259245941 (1h49m52.040179331s) [noise=true , address=104.36.82.5:4840 , piece_id=TAWV7WNIQVSLAIKJLDMP2RIPDN3J5ARSKBFTZVKPRIUS4HS4BQ4Q, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.259673432 (1h49m52.040606817s) [noise=true , address=104.36.82.7:4910 , piece_id=ZD7PQE4NP7QMKW2DA2UF47BDYZNPKDF5AO7SSDUOEDD7U2GYIISQ, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.260261110 (1h49m52.041194499s) [noise=true , address=104.36.82.11:4113 , piece_id=KMSGRHSQCYB77VCEQ7LXL4XSAWEJN6ZXUVINQHWCWSDWQVVY64YQ, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.260716032 (1h49m52.041649416s) [noise=true , address=77.90.31.10:4517 , piece_id=N6FQ6QET5Y6KO7KA7AUDDBKZVRG7UNCQVYDBR4TQP3YHSOO74E6Q, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.262834753 (1h49m52.04376814s) [noise=true , address=77.90.31.16:4733 , piece_id=2J75KCPYIOOC4WM6WPHH4HS2VJMK6MK6ICTUTACP4YLCRXATZBFQ, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.263419790 (1h49m52.044353176s) [noise=true , address=104.36.82.10:4078 , piece_id=V5OP6J7B5K3UFIWDCKWXMTHIB7SG35MI7W75RHWJDYE25N6FZELQ, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.263929602 (1h49m52.044862988s) [noise=true , address=104.36.82.8:4971 , piece_id=PLZKBWZ2S3VCGZFDUYB6JJZRACJY4BSAKH65VPJ4A3TGLRZK7JFQ, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.264484642 (1h49m52.04541803s) [noise=true , address=77.90.31.12:4589 , piece_id=3BSILG2WOW7FGFQAQU64WOIDNBZ7Z6DTLFMRMYT6HATBAZPR4IXA, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.264974774 (1h49m52.04590816s) [noise=true , address=104.36.82.14:4223 , piece_id=J5IUFPDIZNMPDRDDXU25DNMHZVAXL6XSHYM5AU2WEWF24EYPZYNQ, seg_pos={0 0}, upload=1680]: Uploading piece... Jul 26 23:41:36.351658895 (1h49m52.132592281s) [upload=1680]: Next returned nil segment Jul 26 23:41:36.351682811 (1h49m52.132616199s) [upload=1680]: Waiting for error group managing segments... Jul 26 23:41:36.356875043 (1h49m52.137808408s) [noise=true , address=77.90.31.12:4589 , piece_id=3BSILG2WOW7FGFQAQU64WOIDNBZ7Z6DTLFMRMYT6HATBAZPR4IXA, seg_pos={0 0}, upload=1680]: Done uploading piece. err: <nil> Jul 26 23:41:36.357656780 (1h49m52.138590144s) [noise=true , address=77.90.31.14:4661 , piece_id=RQ5BJWUNECNOOTVGJ257N2KHI5FTZ4UQ7RP7HTXLCYNLN64UXT2Q, seg_pos={0 0}, upload=1680]: Done uploading piece. err: <nil> Jul 26 23:41:36.357722689 (1h49m52.13865605s) [noise=true , address=45.91.116.6:4975 , piece_id=WXCTWLZWW23ZIDSORXNWNO7AHTRUWGEDUSY7X66BUHR6ZQSDJTBA, seg_pos={0 0}, upload=1680]: Done uploading piece. err: <nil> Jul 26 23:41:36.821001688 (1h49m52.601935057s) [noise=true , address=45.91.116.5:4298 , piece_id=OEMTYFGTREMSORNMTTAGMQJGUC7HAHAJUI7I5ITNSEFAAUIK653A, seg_pos={0 0}, upload=1680]: Done uploading piece. err: <nil> Jul 26 23:41:38.345059193 (1h49m54.125992561s) [noise=true , address=77.90.31.16:4733 , piece_id=2J75KCPYIOOC4WM6WPHH4HS2VJMK6MK6ICTUTACP4YLCRXATZBFQ, seg_pos={0 0}, upload=1680]: Done uploading piece. err: <nil> Jul 26 23:42:06.187137843 (1h50m21.968071231s) [noise=true , address=104.36.82.13:4185 , piece_id=YUNDQD3US3UDJ2YBTSHFQJN3WGSNJYE6BLHYYK5RJ653M27JUZOA, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:1J4E8bxy3XRU8MmG8Jn3g5DiiBAGs6smUfYSgRea3mbCTZVgDN, address:104.36.82.13:4185): protocol: rpc: dial tcp 104.36.82.13:4185: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.187229901 (1h50m21.968163281s) [noise=true , address=104.36.82.2:4731 , piece_id=FNTV3CVI67HG4BGPO4CGMY2XB5AFFCA5JN7OKYA5TX4D6TSEIKHA, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:1m6CiDtKdTVJSdeLWBbGZh6LS66pnTP8YoVwFpKSccTKoJfok9, address:104.36.82.2:4731): protocol: write tcp 159.223.143.39:60718->104.36.82.2:4731: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.187420240 (1h50m21.968353627s) [noise=true , address=104.36.82.11:4113 , piece_id=KMSGRHSQCYB77VCEQ7LXL4XSAWEJN6ZXUVINQHWCWSDWQVVY64YQ, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:1SQKETWozrz4PcDC3yhy6V5d8Ryo6NNiyWWEkE5xRcmUj9B6UV, address:104.36.82.11:4113): protocol: write tcp 159.223.143.39:38356->104.36.82.11:4113: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.187508196 (1h50m21.968441577s) [noise=true , address=104.36.82.10:4078 , piece_id=V5OP6J7B5K3UFIWDCKWXMTHIB7SG35MI7W75RHWJDYE25N6FZELQ, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:56256->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.187667620 (1h50m21.968601s) [noise=true , address=77.90.31.10:4517 , piece_id=N6FQ6QET5Y6KO7KA7AUDDBKZVRG7UNCQVYDBR4TQP3YHSOO74E6Q, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:129FbXSqxqTJ99BdhfUePzbNvG8db4bxASiXQjSCbkErTaCEk7A, address:77.90.31.10:4517): protocol: write tcp 159.223.143.39:56852->77.90.31.10:4517: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.187734781 (1h50m21.96866815s) [noise=true , address=104.36.82.8:4971 , piece_id=PLZKBWZ2S3VCGZFDUYB6JJZRACJY4BSAKH65VPJ4A3TGLRZK7JFQ, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: write tcp 159.223.143.39:59706->104.36.82.8:4971: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.187798359 (1h50m21.968731739s) [noise=true , address=104.36.82.12:4149 , piece_id=EPG5G6CKA3REHAH73ONGOIDNGFVYXRHBBWCEBB7WL5M66IBPC2KQ, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 159.223.143.39:45102->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.187939106 (1h50m21.968872485s) [noise=true , address=104.36.82.7:4910 , piece_id=ZD7PQE4NP7QMKW2DA2UF47BDYZNPKDF5AO7SSDUOEDD7U2GYIISQ, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 159.223.143.39:34202->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.188017733 (1h50m21.968951116s) [noise=true , address=104.36.82.14:4223 , piece_id=J5IUFPDIZNMPDRDDXU25DNMHZVAXL6XSHYM5AU2WEWF24EYPZYNQ, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:12C59b8knr62ANoUWyaJZDHm2os1tFm3SKL8d34eKsdpZGHYQWE, address:104.36.82.14:4223): protocol: write tcp 159.223.143.39:52750->104.36.82.14:4223: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.188220167 (1h50m21.96915355s) [noise=true , address=104.36.82.14:4221 , piece_id=BAZUCKNRWY6FIMXJXC5SDMZ3NNMGQLKJYVG2ZNXWMXUIAXZD4UKQ, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:129qMN1kX8C3x3H1Epfc4pUbtVGU1xPiZj43QSLByCTPKqwUoJR, address:104.36.82.14:4221): protocol: write tcp 159.223.143.39:36226->104.36.82.14:4221: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.188241038 (1h50m21.969174418s) [noise=true , address=104.36.82.5:4840 , piece_id=TAWV7WNIQVSLAIKJLDMP2RIPDN3J5ARSKBFTZVKPRIUS4HS4BQ4Q, seg_pos={0 0}, upload=1680]: Done uploading piece. err: ecclient: upload failed (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww, address:104.36.82.5:4840): protocol: write tcp 159.223.143.39:50076->104.36.82.5:4840: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:42:06.188300179 (1h50m21.969233568s) [seg_pos={0 0}, upload=1680]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:42:06.214580561 (1h50m21.995513937s) [upload=1680]: Done uploading segments... Jul 26 23:42:06.214612402 (1h50m21.995545781s) [upload=1680]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.03010557s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..............XXX.X....XX.X.X.X.......XXX...XX.....X.XXX....] duration: 20.276322076s FAIL (2025-07-26 23:41:59 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 20.276347987s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [......XX...X...X..XX.X...X..X.......X...X....XX.............] duration: 28.194715909s FAIL (2025-07-26 23:41:52 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.194772421s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.035685682s FAIL (2025-07-26 23:41:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 9) ======LOGS====== 48m40.051376791s) [noise=true , address=77.90.31.10:4530 , piece_id=FJDP7PPG6HBIEE3TMZVCKSXMHBHHXWL3PQEFDSULKPYUZBXLRPYA, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.270467697 (1h48m40.051401084s) [noise=true , address=79.127.205.233:4624 , piece_id=BLM6NYO6D4OAYE72DT73MV2BEL3QKMNHHPZDETULH7UJIQ53GIPA, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.270917792 (1h48m40.051851171s) [noise=true , address=79.127.205.235:4688 , piece_id=B7N5D6L2TIHJE2NCC3T4EW7V6AF2OSOG7JP7FZVOW5KLIB3VHYQA, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.270998428 (1h48m40.051931804s) [noise=true , address=64.49.13.140:4508 , piece_id=4ZVYHRUAFN5SCRH66KURZVVOCA4TDFQATKM6RPTYMYM7NPIVE35A, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.271350958 (1h48m40.052284346s) [noise=true , address=104.36.82.46:4515 , piece_id=R2BJK7TSBUPWOS6R5MZ772RYT5HUFZHMW2Q3MUWAOWXZ5BEZUPTA, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.271473112 (1h48m40.052406516s) [noise=true , address=104.36.82.13:4201 , piece_id=IZNCNXAWBMETWVO5CVPXRJD55OPS325LAQPMMST6JO7Z6WSNE4JQ, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.271698545 (1h48m40.052631923s) [noise=true , address=104.36.82.54:4157 , piece_id=PWD7RXN2ZLEMM4G5EDQTQSDGZ6ZR2522JFOLEUUBVRZTUY4OMQ5Q, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.272011165 (1h48m40.05294454s) [noise=true , address=79.127.205.245:4250 , piece_id=TTAME4MD4D4KDPRZVLK6TY6R5VT257IMQXJTWXQI3NQBCWPXAWHA, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.272019288 (1h48m40.052952671s) [noise=true , address=45.91.116.12:4230 , piece_id=5NJH6XVJXRLRVTXWLTL5RC6K2TPXJDYFFXYPXZ7BNCZ4YZE7Z3TQ, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.272588816 (1h48m40.053522197s) [noise=true , address=79.127.205.234:4610 , piece_id=ZBW5XKDXFIVS6Y6ZTSSJTC4IFWQCNCOM7F2OAHD67K5NZCTDEFAA, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.272612677 (1h48m40.053546052s) [noise=true , address=77.90.31.13:4658 , piece_id=H2HTVHF57ES6KUO5AAZLFBUIBG67K6HTHWGYSHFMR5WYK3F5FMJQ, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.272882654 (1h48m40.053816031s) [noise=true , address=64.49.13.138:4483 , piece_id=IUDDVA7UMZMQ5ZQAQEPPPWL4ULLMCJAPQX6U2UPP5GZ6AOF77OWQ, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.273065664 (1h48m40.053999051s) [noise=true , address=104.36.82.47:4531 , piece_id=OM7UDG2S5GBKWPJCONEX6EGBEZMMFPXXROZQXUZ7S2GCC2MB4VWA, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.273228212 (1h48m40.054161593s) [noise=true , address=45.91.116.14:4194 , piece_id=KAGPM52MUDUX2J5W5OYJJQSTBMVDOZIJ7TZZUA5K3VWUMYWGLCDA, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.273523109 (1h48m40.054456489s) [noise=true , address=77.90.31.18:4809 , piece_id=GWOJI77VCVRVOEJS37LPVAF23P6UZVW6PRIJ4IXW2ZUGUHNDCKFQ, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.273907133 (1h48m40.054840517s) [noise=true , address=104.36.82.56:4260 , piece_id=BEN7QYOJ5CPBFCQ4H5SSKKU67HNCZIFTDBWWLP2OHIBRUPILRWOQ, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.274559142 (1h48m40.055492521s) [noise=true , address=104.36.82.52:4090 , piece_id=AWX7GLYR6CSE4JFUM5F2JSGQLT7VJZGRQNMO7QBSF2EDMI5X5PSQ, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.274953607 (1h48m40.055886985s) [noise=true , address=79.127.219.38:4003 , piece_id=Y2ZCMRSFK476XM5FLL5INNFFPFDUJALFJJLUWQGWOOIVE7ROSQKQ, seg_pos={0 0}, upload=1663]: Uploading piece... Jul 26 23:40:24.350887700 (1h48m40.131821067s) [upload=1663]: Next returned nil segment Jul 26 23:40:24.350903636 (1h48m40.131837011s) [upload=1663]: Waiting for error group managing segments... Jul 26 23:40:24.388668963 (1h48m40.16960233s) [noise=true , address=77.90.31.18:4809 , piece_id=GWOJI77VCVRVOEJS37LPVAF23P6UZVW6PRIJ4IXW2ZUGUHNDCKFQ, seg_pos={0 0}, upload=1663]: Done uploading piece. err: <nil> Jul 26 23:40:24.756826310 (1h48m40.537759674s) [noise=true , address=79.127.219.38:4003 , piece_id=Y2ZCMRSFK476XM5FLL5INNFFPFDUJALFJJLUWQGWOOIVE7ROSQKQ, seg_pos={0 0}, upload=1663]: Done uploading piece. err: <nil> Jul 26 23:40:24.840401438 (1h48m40.621334821s) [noise=true , address=79.127.205.245:4250 , piece_id=TTAME4MD4D4KDPRZVLK6TY6R5VT257IMQXJTWXQI3NQBCWPXAWHA, seg_pos={0 0}, upload=1663]: Done uploading piece. err: <nil> Jul 26 23:40:24.863377794 (1h48m40.644311163s) [noise=true , address=79.127.205.235:4688 , piece_id=B7N5D6L2TIHJE2NCC3T4EW7V6AF2OSOG7JP7FZVOW5KLIB3VHYQA, seg_pos={0 0}, upload=1663]: Done uploading piece. err: <nil> Jul 26 23:40:25.073974314 (1h48m40.854907696s) [noise=true , address=79.127.205.233:4624 , piece_id=BLM6NYO6D4OAYE72DT73MV2BEL3QKMNHHPZDETULH7UJIQ53GIPA, seg_pos={0 0}, upload=1663]: Done uploading piece. err: <nil> Jul 26 23:40:25.159720016 (1h48m40.940653385s) [noise=true , address=79.127.205.234:4610 , piece_id=ZBW5XKDXFIVS6Y6ZTSSJTC4IFWQCNCOM7F2OAHD67K5NZCTDEFAA, seg_pos={0 0}, upload=1663]: Done uploading piece. err: <nil> Jul 26 23:40:26.227989914 (1h48m42.00892328s) [noise=true , address=64.49.13.140:4508 , piece_id=4ZVYHRUAFN5SCRH66KURZVVOCA4TDFQATKM6RPTYMYM7NPIVE35A, seg_pos={0 0}, upload=1663]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 23:40:28.218665173 (1h48m43.999598541s) [noise=true , address=77.90.31.13:4658 , piece_id=H2HTVHF57ES6KUO5AAZLFBUIBG67K6HTHWGYSHFMR5WYK3F5FMJQ, seg_pos={0 0}, upload=1663]: Done uploading piece. err: <nil> Jul 26 23:40:29.666636360 (1h48m45.447569751s) [noise=true , address=45.91.116.14:4194 , piece_id=KAGPM52MUDUX2J5W5OYJJQSTBMVDOZIJ7TZZUA5K3VWUMYWGLCDA, seg_pos={0 0}, upload=1663]: Done uploading piece. err: ecclient: upload failed (node:19mdk9cXDR79ZXC4n6rMXFUhpwxMPXFpkPA14ymY24RK1jKkmd, address:45.91.116.14:4194): protocol: storage node overloaded, request limit: 2000 Jul 26 23:40:30.524754720 (1h48m46.305688103s) [noise=true , address=45.91.116.12:4230 , piece_id=5NJH6XVJXRLRVTXWLTL5RC6K2TPXJDYFFXYPXZ7BNCZ4YZE7Z3TQ, seg_pos={0 0}, upload=1663]: Done uploading piece. err: <nil> Jul 26 23:40:30.866554622 (1h48m46.647488005s) [noise=true , address=64.49.13.138:4483 , piece_id=IUDDVA7UMZMQ5ZQAQEPPPWL4ULLMCJAPQX6U2UPP5GZ6AOF77OWQ, seg_pos={0 0}, upload=1663]: Done uploading piece. err: <nil> Jul 26 23:41:24.111643473 (1h49m39.892576852s) [noise=true , address=104.36.82.13:4201 , piece_id=IZNCNXAWBMETWVO5CVPXRJD55OPS325LAQPMMST6JO7Z6WSNE4JQ, seg_pos={0 0}, upload=1663]: Done uploading piece. err: ecclient: upload failed (node:12VTXMHwWTyB6H6RRCAk7uupYBGhhDcyngUBx7cgUjjYA1vegL6, address:104.36.82.13:4201): protocol: write tcp 159.223.143.39:34040->104.36.82.13:4201: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:24.111808870 (1h49m39.892742251s) [noise=true , address=104.36.82.47:4531 , piece_id=OM7UDG2S5GBKWPJCONEX6EGBEZMMFPXXROZQXUZ7S2GCC2MB4VWA, seg_pos={0 0}, upload=1663]: Done uploading piece. err: ecclient: upload failed (node:12LZprfyqfD2YUwgAarzyjbgHEb6jMhAMAday5C45xk3j9RUS4u, address:104.36.82.47:4531): protocol: rpc: dial tcp 104.36.82.47:4531: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:24.111893200 (1h49m39.892826576s) [noise=true , address=104.36.82.56:4260 , piece_id=BEN7QYOJ5CPBFCQ4H5SSKKU67HNCZIFTDBWWLP2OHIBRUPILRWOQ, seg_pos={0 0}, upload=1663]: Done uploading piece. err: ecclient: upload failed (node:1NjKYY18P7vTepAAujSj8Gep3STrHBdAdwaQ4VS6noje1oqSCH, address:104.36.82.56:4260): protocol: rpc: dial tcp 104.36.82.56:4260: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:24.111949764 (1h49m39.89288314s) [noise=true , address=104.36.82.52:4090 , piece_id=AWX7GLYR6CSE4JFUM5F2JSGQLT7VJZGRQNMO7QBSF2EDMI5X5PSQ, seg_pos={0 0}, upload=1663]: Done uploading piece. err: ecclient: upload failed (node:12UUnoD9p5FC7rq9fMSzu1phkFpsnPsu4UYxC59MhpMjJ6wUC6F, address:104.36.82.52:4090): protocol: rpc: dial tcp 104.36.82.52:4090: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:24.112026515 (1h49m39.89295988s) [noise=true , address=77.90.31.10:4530 , piece_id=FJDP7PPG6HBIEE3TMZVCKSXMHBHHXWL3PQEFDSULKPYUZBXLRPYA, seg_pos={0 0}, upload=1663]: Done uploading piece. err: ecclient: upload failed (node:127c4HGgJu3z16vxq7TYZydCUa86FhCc3TmoPZoheVeqTbnJfne, address:77.90.31.10:4530): protocol: expected piece hash; context deadline exceeded; EOF Jul 26 23:41:24.112235751 (1h49m39.893169194s) [noise=true , address=104.36.82.54:4157 , piece_id=PWD7RXN2ZLEMM4G5EDQTQSDGZ6ZR2522JFOLEUUBVRZTUY4OMQ5Q, seg_pos={0 0}, upload=1663]: Done uploading piece. err: ecclient: upload failed (node:1ZrzADk2mKG4R9tjxUKjQeJvb4WmfjLksGFyvstHRfYbBC7XNz, address:104.36.82.54:4157): protocol: rpc: dial tcp 104.36.82.54:4157: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:24.112307985 (1h49m39.893241368s) [noise=true , address=104.36.82.46:4515 , piece_id=R2BJK7TSBUPWOS6R5MZ772RYT5HUFZHMW2Q3MUWAOWXZ5BEZUPTA, seg_pos={0 0}, upload=1663]: Done uploading piece. err: ecclient: upload failed (node:1iDufTPd97E4DAupki6D6DxNZS8S1gKwynNBYynsauhbw5waUs, address:104.36.82.46:4515): protocol: write tcp 159.223.143.39:46716->104.36.82.46:4515: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:41:24.112360351 (1h49m39.893293735s) [seg_pos={0 0}, upload=1663]: Done waiting for segment. successful: 9 optimal: 14 errs: <nil> Jul 26 23:41:24.145267611 (1h49m39.926200973s) [upload=1663]: Done uploading segments... Jul 26 23:41:24.145290823 (1h49m39.926224184s) [upload=1663]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 9) Duration: 1m0.035706309s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 44 out of 60 is failed (threshold=3) [..........X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.026249983s FAIL (2025-07-26 23:41:19 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.026273243s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.......XX...X...X..XX.X...X..X.......X...X....XX............] duration: 43.132907093s FAIL (2025-07-26 23:41:18 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 43.132947676s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [...............XXX.X....XX.X.X.X.......XXX...XX.....X.XXX...] duration: 9.113012538s FAIL (2025-07-26 23:41:07 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 9.113040771s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [................XXX.X....XX.X.X.X.......XXX...XX.....X.XXX..] duration: 4.292889632s FAIL (2025-07-26 23:40:46 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 4.293007942s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [.................XXX.X....XX.X.X.X.......XXX...XX.....X.XXX.] duration: 28.068937431s FAIL (2025-07-26 23:40:38 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 28.068979967s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [........XX...X...X..XX.X...X..X.......X...X....XX...........] duration: 40.224662823s FAIL (2025-07-26 23:40:35 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 40.224677536s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.] duration: 1.76924134s FAIL (2025-07-26 23:40:25 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== : Starting upload Jul 26 23:39:53.651659456 (1h48m9.432592821s) [upload=1656]: Done starting upload Jul 26 23:39:53.651795715 (1h48m9.432729094s) [upload=1656]: Uploading segments... Jul 26 23:39:53.652022529 (1h48m9.432955897s) [upload=1656]: Got next segment. Inline: false Jul 26 23:39:53.652177070 (1h48m9.433110436s) [upload=1656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:39:53.652184134 (1h48m9.433117509s) [upload=1656]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:39:53.760344617 (1h48m9.541278013s) [seg_pos={0 0}, upload=1656]: Begin upload segment... Jul 26 23:39:53.760424673 (1h48m9.541358039s) [seg_pos={0 0}, upload=1656]: Done begin upload segment. Jul 26 23:39:53.760459139 (1h48m9.541392526s) [noise=true , address=45.91.116.4:4937 , piece_id=DFS3LWY7DKFESCASK5OWX3XK5W4SMUMT37B2RFPAVVUJZKRWREQQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.760511419 (1h48m9.5414448s) [noise=true , address=104.36.82.4:4801 , piece_id=YVWA5KTVT5QSSOGGOF7XWQKJ2LWK3ECGMPIQJFHOYXC26YWMVVJQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.761043421 (1h48m9.541976797s) [noise=true , address=45.91.116.5:4298 , piece_id=V72G2DRY5TICE46L756GPFOSUPWEZXML7EESHRTDD2U6Q5JWF2PQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.761155851 (1h48m9.542089232s) [noise=true , address=104.36.82.6:4873 , piece_id=WVB7T22GHC62S46HSQBRE4HED5SIYWYL7JKK4X6I5GXPCKP5TG4Q, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.761542565 (1h48m9.542475946s) [noise=true , address=45.91.116.6:4974 , piece_id=3AHWSMMGY43Y5A4NBDCK4WBVVQWVTLPNLQXKNQLHA4CYXFSLYM2A, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.761742109 (1h48m9.542675495s) [noise=true , address=45.91.116.12:4225 , piece_id=WQ3PPLNJY4VUQENA74PWMF6IY3RREIQQKE6JWCZBTWYZIL26SBPQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.762101713 (1h48m9.543035094s) [noise=true , address=45.91.116.12:4226 , piece_id=ZWL2YQKJQ6WEJP7TZOGJAXMYFW7Y32OPHYQBLEW5BUCFKBKOMYAQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.762304875 (1h48m9.543238262s) [noise=true , address=45.91.116.13:4155 , piece_id=PTDKP5JDWN3LUL2MLU36QKOD6PFLVNX3UGHK6CYNLYX35YEILFYA, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.762586015 (1h48m9.543519392s) [noise=true , address=45.91.116.10:4081 , piece_id=NGXCCV4INYZSBSMOWYIVWN4UCZXK5PKLTTHMKGDEJD6VGWJIDSSQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.762908490 (1h48m9.543841879s) [noise=true , address=104.36.82.11:4115 , piece_id=ETQQZVQUMKRV6YMFRFK4EI2KEL3MB7DBMSZ3TUZW3HORULXR37NQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.763217573 (1h48m9.544150957s) [noise=true , address=104.36.82.5:4838 , piece_id=FNJVB7YFWDKO7RFFUWNTF6W4UNQTW7GBFCDSMKCNBHKQU3GXBBXA, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.763475029 (1h48m9.544408418s) [noise=true , address=104.36.82.15:4257 , piece_id=2DOVYVWWTRJFW6MUN6ESVEI2AB5CBLN4OPNET7C4OACFIFCP7CQQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.763719412 (1h48m9.544652792s) [noise=true , address=104.36.82.2:4730 , piece_id=JZ32XYPAJENIQKUIRSBWWZHV5DQMZPMYGWBOZV5OIL4OUVKQDKWQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.763976761 (1h48m9.544910146s) [noise=true , address=104.36.82.7:4909 , piece_id=5CUKBI35ADL5QLJXQQP2V4EHLVHKLWOCXQWDSXIGZL6ZMWVOM6NA, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.764165834 (1h48m9.54509922s) [noise=true , address=104.36.82.7:4912 , piece_id=GS3HZK2674DDXRTAVFFWBMVIO5MIRC4PJVCFKD5MQXDIBLXTXC2A, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.764445958 (1h48m9.545379346s) [noise=true , address=104.36.82.2:4729 , piece_id=WLJPYXF5K6IGG47OEUCDUDP4SHVOOTMXG22GTNDJMAF4JFMFOX3A, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.857635650 (1h48m9.638569016s) [upload=1656]: Next returned nil segment Jul 26 23:39:53.857653813 (1h48m9.638587183s) [upload=1656]: Waiting for error group managing segments... Jul 26 23:39:53.861881868 (1h48m9.642815249s) [noise=true , address=45.91.116.13:4155 , piece_id=PTDKP5JDWN3LUL2MLU36QKOD6PFLVNX3UGHK6CYNLYX35YEILFYA, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.861935675 (1h48m9.642869038s) [noise=true , address=45.91.116.4:4937 , piece_id=DFS3LWY7DKFESCASK5OWX3XK5W4SMUMT37B2RFPAVVUJZKRWREQQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.862515297 (1h48m9.643448675s) [noise=true , address=45.91.116.10:4081 , piece_id=NGXCCV4INYZSBSMOWYIVWN4UCZXK5PKLTTHMKGDEJD6VGWJIDSSQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.862570367 (1h48m9.643503736s) [noise=true , address=45.91.116.12:4226 , piece_id=ZWL2YQKJQ6WEJP7TZOGJAXMYFW7Y32OPHYQBLEW5BUCFKBKOMYAQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.863057776 (1h48m9.643991146s) [noise=true , address=45.91.116.6:4974 , piece_id=3AHWSMMGY43Y5A4NBDCK4WBVVQWVTLPNLQXKNQLHA4CYXFSLYM2A, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.866234319 (1h48m9.6471677s) [noise=true , address=45.91.116.12:4225 , piece_id=WQ3PPLNJY4VUQENA74PWMF6IY3RREIQQKE6JWCZBTWYZIL26SBPQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.985553629 (1h48m9.766486998s) [noise=true , address=45.91.116.5:4298 , piece_id=V72G2DRY5TICE46L756GPFOSUPWEZXML7EESHRTDD2U6Q5JWF2PQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:40:23.614501940 (1h48m39.395435321s) [noise=true , address=104.36.82.6:4873 , piece_id=WVB7T22GHC62S46HSQBRE4HED5SIYWYL7JKK4X6I5GXPCKP5TG4Q, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:46296->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.614816596 (1h48m39.395749978s) [noise=true , address=104.36.82.2:4729 , piece_id=WLJPYXF5K6IGG47OEUCDUDP4SHVOOTMXG22GTNDJMAF4JFMFOX3A, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.614954894 (1h48m39.395888279s) [noise=true , address=104.36.82.2:4730 , piece_id=JZ32XYPAJENIQKUIRSBWWZHV5DQMZPMYGWBOZV5OIL4OUVKQDKWQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615058497 (1h48m39.39599188s) [noise=true , address=104.36.82.7:4909 , piece_id=5CUKBI35ADL5QLJXQQP2V4EHLVHKLWOCXQWDSXIGZL6ZMWVOM6NA, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615118572 (1h48m39.396051953s) [noise=true , address=104.36.82.11:4115 , piece_id=ETQQZVQUMKRV6YMFRFK4EI2KEL3MB7DBMSZ3TUZW3HORULXR37NQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615198214 (1h48m39.396131594s) [noise=true , address=104.36.82.5:4838 , piece_id=FNJVB7YFWDKO7RFFUWNTF6W4UNQTW7GBFCDSMKCNBHKQU3GXBBXA, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615286224 (1h48m39.3962196s) [noise=true , address=104.36.82.15:4257 , piece_id=2DOVYVWWTRJFW6MUN6ESVEI2AB5CBLN4OPNET7C4OACFIFCP7CQQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615304725 (1h48m39.396238092s) [noise=true , address=104.36.82.7:4912 , piece_id=GS3HZK2674DDXRTAVFFWBMVIO5MIRC4PJVCFKD5MQXDIBLXTXC2A, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615342159 (1h48m39.396275535s) [noise=true , address=104.36.82.4:4801 , piece_id=YVWA5KTVT5QSSOGGOF7XWQKJ2LWK3ECGMPIQJFHOYXC26YWMVVJQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615382082 (1h48m39.396315457s) [seg_pos={0 0}, upload=1656]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:40:23.662236247 (1h48m39.443169685s) [upload=1656]: Done uploading segments... Jul 26 23:40:23.662284995 (1h48m39.443218371s) [upload=1656]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 1.769266308s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [X.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.028528971s FAIL (2025-07-26 23:40:24 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== Jul 26 23:39:24.055606865 (1h47m39.836540231s) [seg_pos={0 0}, upload=1648]: Done begin upload segment. Jul 26 23:39:24.055640127 (1h47m39.836573502s) [noise=true , address=104.36.82.55:4207 , piece_id=YVFUZ2LA4OA3AFJGEY7X3ZVRLZZXM3WNFSFVBOQ4BT4TJOF2OQEA, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.055660702 (1h47m39.836594087s) [noise=true , address=104.36.82.46:4492 , piece_id=JMGB57YRCYCU6ADCVVDG5ELOEZMMRTB3BF7Q3ODKOHHDAH53BK4A, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.056101217 (1h47m39.837034659s) [noise=true , address=104.36.82.52:4092 , piece_id=SM24GEERL7EHLUPKZ422HP75G3U5DGV76KBO27K2TZJESKA7JOKA, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.056216367 (1h47m39.837149744s) [noise=true , address=79.127.201.215:4353 , piece_id=2BMTPA7JVRDZCZUXEN46YW2GV4KJP4VUTXXWCKSADVFBS43V75PA, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.056443546 (1h47m39.837376985s) [noise=true , address=104.36.82.22:4537 , piece_id=7STK4LSTI6IWOWREQNSKQVH5ZGSQU3IBCJJFQQTPITK6HKB4GMGQ, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.056785677 (1h47m39.837719055s) [noise=true , address=79.127.205.235:4691 , piece_id=DGWB6WXMIOYMYYD6S3KH52UZMQVXNUQB5CGODP47O4IM77MJHYUQ, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.059670543 (1h47m39.840603927s) [noise=true , address=77.90.31.12:4614 , piece_id=SJ5V3WFJVN3376OE4AIOLRUUWTAGJ3IHZM2WHBQ6D5PMLFMSA4LA, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.060225984 (1h47m39.84115937s) [noise=true , address=79.127.205.234:4618 , piece_id=Z2QWR75CR7PO6LNLQKV62H4WQGAI26BKLTZN3GSXE2PT5XLHG6LA, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.060778266 (1h47m39.841711651s) [noise=true , address=45.91.116.11:4121 , piece_id=QHOLWAUP4JJ2P4FSHOPIXQJFR2J67LZIKJJM4W7DV6WJG6JMLDIQ, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.061271482 (1h47m39.842204864s) [noise=true , address=79.127.205.233:4640 , piece_id=22M2A6452J6XA3JFCE3C6GGYNXHDLT7EYTBDIPICAGTY2DDSKFOA, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.061704848 (1h47m39.842638234s) [noise=true , address=77.90.31.21:4912 , piece_id=DKHCR66UDLX3Z4NKIGQLMA5XMAKPX5ASLRZKND3LKCRZBVGQGUNA, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.062095731 (1h47m39.843029113s) [noise=true , address=104.36.82.54:4158 , piece_id=JGI64PYCJMQKGBGJEPTEGEWSO4B2QADCF5O34Y77VBVMYL26472Q, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.062459584 (1h47m39.843392966s) [noise=true , address=45.91.116.5:4310 , piece_id=LZ4V7S4QBRX3AGHCTBNHVFCJWYSRTHVASBCLNRW6UKJZRZMPFD2Q, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.062862004 (1h47m39.843795387s) [noise=true , address=45.91.116.9:4273 , piece_id=QPA3YEG3VWDEDXKJUHAA77TBPV7VIEQKWLLIIT4NRX7JETDSSOOQ, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.063252426 (1h47m39.844185809s) [noise=true , address=77.90.31.11:4558 , piece_id=VBLQN6C6F5NJNUX533PZ6XG2Q74352NBWICFCYBSJA53VC4GXA4A, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.063639960 (1h47m39.844573341s) [noise=true , address=79.127.219.38:4015 , piece_id=DHD4PWLFEUT5Y27KTMGFCAUCHQNSM4VXJZLDS27LMKXMPQRF7V2Q, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.064095102 (1h47m39.845028485s) [noise=true , address=79.127.205.245:4249 , piece_id=YUDB257EPXWX5OLB62WWWVXQEPYYR2RZRW6PHUCJXCPA6GKFZXWA, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.064477662 (1h47m39.84541103s) [noise=true , address=104.36.82.47:4530 , piece_id=AWIXTDPD44XGNJ4SGIL5GLUQDBEWZQLR4RMMLQUSPKRFGXBRG7ZA, seg_pos={0 0}, upload=1648]: Uploading piece... Jul 26 23:39:24.125017691 (1h47m39.905951054s) [upload=1648]: Next returned nil segment Jul 26 23:39:24.125039909 (1h47m39.905973295s) [upload=1648]: Waiting for error group managing segments... Jul 26 23:39:24.129407680 (1h47m39.910341068s) [noise=true , address=79.127.219.38:4015 , piece_id=DHD4PWLFEUT5Y27KTMGFCAUCHQNSM4VXJZLDS27LMKXMPQRF7V2Q, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:39:24.168785826 (1h47m39.949719201s) [noise=true , address=77.90.31.12:4614 , piece_id=SJ5V3WFJVN3376OE4AIOLRUUWTAGJ3IHZM2WHBQ6D5PMLFMSA4LA, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:39:24.657921679 (1h47m40.438855055s) [noise=true , address=79.127.205.233:4640 , piece_id=22M2A6452J6XA3JFCE3C6GGYNXHDLT7EYTBDIPICAGTY2DDSKFOA, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:39:24.662741356 (1h47m40.443674723s) [noise=true , address=79.127.205.234:4618 , piece_id=Z2QWR75CR7PO6LNLQKV62H4WQGAI26BKLTZN3GSXE2PT5XLHG6LA, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:39:24.991068588 (1h47m40.772001964s) [noise=true , address=79.127.205.235:4691 , piece_id=DGWB6WXMIOYMYYD6S3KH52UZMQVXNUQB5CGODP47O4IM77MJHYUQ, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:39:27.062691589 (1h47m42.843624956s) [noise=true , address=45.91.116.11:4121 , piece_id=QHOLWAUP4JJ2P4FSHOPIXQJFR2J67LZIKJJM4W7DV6WJG6JMLDIQ, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:39:29.507782069 (1h47m45.288715437s) [noise=true , address=79.127.205.245:4249 , piece_id=YUDB257EPXWX5OLB62WWWVXQEPYYR2RZRW6PHUCJXCPA6GKFZXWA, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:39:30.163105689 (1h47m45.944039054s) [noise=true , address=45.91.116.9:4273 , piece_id=QPA3YEG3VWDEDXKJUHAA77TBPV7VIEQKWLLIIT4NRX7JETDSSOOQ, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:39:31.510364639 (1h47m47.291298001s) [noise=true , address=45.91.116.5:4310 , piece_id=LZ4V7S4QBRX3AGHCTBNHVFCJWYSRTHVASBCLNRW6UKJZRZMPFD2Q, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:39:36.449125481 (1h47m52.230058849s) [noise=true , address=79.127.201.215:4353 , piece_id=2BMTPA7JVRDZCZUXEN46YW2GV4KJP4VUTXXWCKSADVFBS43V75PA, seg_pos={0 0}, upload=1648]: Done uploading piece. err: <nil> Jul 26 23:40:02.128987889 (1h48m17.909921269s) [noise=true , address=77.90.31.11:4558 , piece_id=VBLQN6C6F5NJNUX533PZ6XG2Q74352NBWICFCYBSJA53VC4GXA4A, seg_pos={0 0}, upload=1648]: Done uploading piece. err: ecclient: upload failed (node:1mGEP1n1icC2gEkyRVy6BeM4vsrgqYWaQeB5rUtxjrWQjyFhTS, address:77.90.31.11:4558): protocol: storage node overloaded, request limit: 2000 Jul 26 23:40:23.990505710 (1h48m39.771439098s) [noise=true , address=104.36.82.52:4092 , piece_id=SM24GEERL7EHLUPKZ422HP75G3U5DGV76KBO27K2TZJESKA7JOKA, seg_pos={0 0}, upload=1648]: Done uploading piece. err: ecclient: upload failed (node:12wYaUu5QpFU8T4mZt1iqeHqVJKJ5CyVW5vkeTWf1a8G3pLsx3P, address:104.36.82.52:4092): protocol: write tcp 159.223.143.39:37472->104.36.82.52:4092: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.990732371 (1h48m39.77166575s) [noise=true , address=104.36.82.54:4158 , piece_id=JGI64PYCJMQKGBGJEPTEGEWSO4B2QADCF5O34Y77VBVMYL26472Q, seg_pos={0 0}, upload=1648]: Done uploading piece. err: ecclient: upload failed (node:1vJZrSvPvwebw9V2tTbvkPEmLeLjryBUnq9TL6NwdPsHU9xcq2, address:104.36.82.54:4158): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:40:23.991012588 (1h48m39.771945962s) [noise=true , address=104.36.82.46:4492 , piece_id=JMGB57YRCYCU6ADCVVDG5ELOEZMMRTB3BF7Q3ODKOHHDAH53BK4A, seg_pos={0 0}, upload=1648]: Done uploading piece. err: ecclient: upload failed (node:1roa5GGYmqCX2AxSrqFgqyV2CtBEYZavuGTseiwDy4pjkT6bip, address:104.36.82.46:4492): protocol: write tcp 159.223.143.39:56314->104.36.82.46:4492: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.991174828 (1h48m39.772108201s) [noise=true , address=104.36.82.47:4530 , piece_id=AWIXTDPD44XGNJ4SGIL5GLUQDBEWZQLR4RMMLQUSPKRFGXBRG7ZA, seg_pos={0 0}, upload=1648]: Done uploading piece. err: ecclient: upload failed (node:1WVxMFx9jm6CT7GcXg2qgq3Be9UPFf1KZKssqufSk2ebdBBmxR, address:104.36.82.47:4530): protocol: rpc: dial tcp 104.36.82.47:4530: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.991224739 (1h48m39.772158108s) [noise=true , address=104.36.82.55:4207 , piece_id=YVFUZ2LA4OA3AFJGEY7X3ZVRLZZXM3WNFSFVBOQ4BT4TJOF2OQEA, seg_pos={0 0}, upload=1648]: Done uploading piece. err: ecclient: upload failed (node:121hXqdHirzNQq5H1ynCC3bboxCyYpeRfBbZR4zPQCPiW2ebbey, address:104.36.82.55:4207): protocol: write tcp 159.223.143.39:59898->104.36.82.55:4207: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.991288266 (1h48m39.772221641s) [noise=true , address=104.36.82.22:4537 , piece_id=7STK4LSTI6IWOWREQNSKQVH5ZGSQU3IBCJJFQQTPITK6HKB4GMGQ, seg_pos={0 0}, upload=1648]: Done uploading piece. err: ecclient: upload failed (node:12U3QSKhAmoaakTJRReBtoRpzbvzYdibd1xjsEVuU8LD29Fcu1y, address:104.36.82.22:4537): protocol: write tcp 159.223.143.39:58114->104.36.82.22:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.991443004 (1h48m39.77237637s) [noise=true , address=77.90.31.21:4912 , piece_id=DKHCR66UDLX3Z4NKIGQLMA5XMAKPX5ASLRZKND3LKCRZBVGQGUNA, seg_pos={0 0}, upload=1648]: Done uploading piece. err: ecclient: upload failed (node:12SFktrdgGxzEZue49XWkNxVZfrpHA73M2H3hRX7NXnvwVnvbv1, address:77.90.31.21:4912): protocol: context deadline exceeded; context deadline exceeded Jul 26 23:40:23.991479303 (1h48m39.772412688s) [seg_pos={0 0}, upload=1648]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:40:24.017354981 (1h48m39.798288347s) [upload=1648]: Done uploading segments... Jul 26 23:40:24.017385598 (1h48m39.798318996s) [upload=1648]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.028554508s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 39 out of 60 is failed (threshold=3) [.....................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.050078305s FAIL (2025-07-26 23:40:23 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 7) ======LOGS====== : Starting upload Jul 26 23:39:53.651659456 (1h48m9.432592821s) [upload=1656]: Done starting upload Jul 26 23:39:53.651795715 (1h48m9.432729094s) [upload=1656]: Uploading segments... Jul 26 23:39:53.652022529 (1h48m9.432955897s) [upload=1656]: Got next segment. Inline: false Jul 26 23:39:53.652177070 (1h48m9.433110436s) [upload=1656]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:39:53.652184134 (1h48m9.433117509s) [upload=1656]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:39:53.760344617 (1h48m9.541278013s) [seg_pos={0 0}, upload=1656]: Begin upload segment... Jul 26 23:39:53.760424673 (1h48m9.541358039s) [seg_pos={0 0}, upload=1656]: Done begin upload segment. Jul 26 23:39:53.760459139 (1h48m9.541392526s) [noise=true , address=45.91.116.4:4937 , piece_id=DFS3LWY7DKFESCASK5OWX3XK5W4SMUMT37B2RFPAVVUJZKRWREQQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.760511419 (1h48m9.5414448s) [noise=true , address=104.36.82.4:4801 , piece_id=YVWA5KTVT5QSSOGGOF7XWQKJ2LWK3ECGMPIQJFHOYXC26YWMVVJQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.761043421 (1h48m9.541976797s) [noise=true , address=45.91.116.5:4298 , piece_id=V72G2DRY5TICE46L756GPFOSUPWEZXML7EESHRTDD2U6Q5JWF2PQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.761155851 (1h48m9.542089232s) [noise=true , address=104.36.82.6:4873 , piece_id=WVB7T22GHC62S46HSQBRE4HED5SIYWYL7JKK4X6I5GXPCKP5TG4Q, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.761542565 (1h48m9.542475946s) [noise=true , address=45.91.116.6:4974 , piece_id=3AHWSMMGY43Y5A4NBDCK4WBVVQWVTLPNLQXKNQLHA4CYXFSLYM2A, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.761742109 (1h48m9.542675495s) [noise=true , address=45.91.116.12:4225 , piece_id=WQ3PPLNJY4VUQENA74PWMF6IY3RREIQQKE6JWCZBTWYZIL26SBPQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.762101713 (1h48m9.543035094s) [noise=true , address=45.91.116.12:4226 , piece_id=ZWL2YQKJQ6WEJP7TZOGJAXMYFW7Y32OPHYQBLEW5BUCFKBKOMYAQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.762304875 (1h48m9.543238262s) [noise=true , address=45.91.116.13:4155 , piece_id=PTDKP5JDWN3LUL2MLU36QKOD6PFLVNX3UGHK6CYNLYX35YEILFYA, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.762586015 (1h48m9.543519392s) [noise=true , address=45.91.116.10:4081 , piece_id=NGXCCV4INYZSBSMOWYIVWN4UCZXK5PKLTTHMKGDEJD6VGWJIDSSQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.762908490 (1h48m9.543841879s) [noise=true , address=104.36.82.11:4115 , piece_id=ETQQZVQUMKRV6YMFRFK4EI2KEL3MB7DBMSZ3TUZW3HORULXR37NQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.763217573 (1h48m9.544150957s) [noise=true , address=104.36.82.5:4838 , piece_id=FNJVB7YFWDKO7RFFUWNTF6W4UNQTW7GBFCDSMKCNBHKQU3GXBBXA, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.763475029 (1h48m9.544408418s) [noise=true , address=104.36.82.15:4257 , piece_id=2DOVYVWWTRJFW6MUN6ESVEI2AB5CBLN4OPNET7C4OACFIFCP7CQQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.763719412 (1h48m9.544652792s) [noise=true , address=104.36.82.2:4730 , piece_id=JZ32XYPAJENIQKUIRSBWWZHV5DQMZPMYGWBOZV5OIL4OUVKQDKWQ, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.763976761 (1h48m9.544910146s) [noise=true , address=104.36.82.7:4909 , piece_id=5CUKBI35ADL5QLJXQQP2V4EHLVHKLWOCXQWDSXIGZL6ZMWVOM6NA, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.764165834 (1h48m9.54509922s) [noise=true , address=104.36.82.7:4912 , piece_id=GS3HZK2674DDXRTAVFFWBMVIO5MIRC4PJVCFKD5MQXDIBLXTXC2A, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.764445958 (1h48m9.545379346s) [noise=true , address=104.36.82.2:4729 , piece_id=WLJPYXF5K6IGG47OEUCDUDP4SHVOOTMXG22GTNDJMAF4JFMFOX3A, seg_pos={0 0}, upload=1656]: Uploading piece... Jul 26 23:39:53.857635650 (1h48m9.638569016s) [upload=1656]: Next returned nil segment Jul 26 23:39:53.857653813 (1h48m9.638587183s) [upload=1656]: Waiting for error group managing segments... Jul 26 23:39:53.861881868 (1h48m9.642815249s) [noise=true , address=45.91.116.13:4155 , piece_id=PTDKP5JDWN3LUL2MLU36QKOD6PFLVNX3UGHK6CYNLYX35YEILFYA, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.861935675 (1h48m9.642869038s) [noise=true , address=45.91.116.4:4937 , piece_id=DFS3LWY7DKFESCASK5OWX3XK5W4SMUMT37B2RFPAVVUJZKRWREQQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.862515297 (1h48m9.643448675s) [noise=true , address=45.91.116.10:4081 , piece_id=NGXCCV4INYZSBSMOWYIVWN4UCZXK5PKLTTHMKGDEJD6VGWJIDSSQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.862570367 (1h48m9.643503736s) [noise=true , address=45.91.116.12:4226 , piece_id=ZWL2YQKJQ6WEJP7TZOGJAXMYFW7Y32OPHYQBLEW5BUCFKBKOMYAQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.863057776 (1h48m9.643991146s) [noise=true , address=45.91.116.6:4974 , piece_id=3AHWSMMGY43Y5A4NBDCK4WBVVQWVTLPNLQXKNQLHA4CYXFSLYM2A, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.866234319 (1h48m9.6471677s) [noise=true , address=45.91.116.12:4225 , piece_id=WQ3PPLNJY4VUQENA74PWMF6IY3RREIQQKE6JWCZBTWYZIL26SBPQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:39:53.985553629 (1h48m9.766486998s) [noise=true , address=45.91.116.5:4298 , piece_id=V72G2DRY5TICE46L756GPFOSUPWEZXML7EESHRTDD2U6Q5JWF2PQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: <nil> Jul 26 23:40:23.614501940 (1h48m39.395435321s) [noise=true , address=104.36.82.6:4873 , piece_id=WVB7T22GHC62S46HSQBRE4HED5SIYWYL7JKK4X6I5GXPCKP5TG4Q, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:46296->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.614816596 (1h48m39.395749978s) [noise=true , address=104.36.82.2:4729 , piece_id=WLJPYXF5K6IGG47OEUCDUDP4SHVOOTMXG22GTNDJMAF4JFMFOX3A, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.614954894 (1h48m39.395888279s) [noise=true , address=104.36.82.2:4730 , piece_id=JZ32XYPAJENIQKUIRSBWWZHV5DQMZPMYGWBOZV5OIL4OUVKQDKWQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom, address:104.36.82.2:4730): protocol: rpc: dial tcp 104.36.82.2:4730: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615058497 (1h48m39.39599188s) [noise=true , address=104.36.82.7:4909 , piece_id=5CUKBI35ADL5QLJXQQP2V4EHLVHKLWOCXQWDSXIGZL6ZMWVOM6NA, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12DaDrwzfrbz1yNwt4CpyZq7hADh5KQpJZkVmev8ukpF9wcXK2x, address:104.36.82.7:4909): protocol: rpc: dial tcp 104.36.82.7:4909: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615118572 (1h48m39.396051953s) [noise=true , address=104.36.82.11:4115 , piece_id=ETQQZVQUMKRV6YMFRFK4EI2KEL3MB7DBMSZ3TUZW3HORULXR37NQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615198214 (1h48m39.396131594s) [noise=true , address=104.36.82.5:4838 , piece_id=FNJVB7YFWDKO7RFFUWNTF6W4UNQTW7GBFCDSMKCNBHKQU3GXBBXA, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615286224 (1h48m39.3962196s) [noise=true , address=104.36.82.15:4257 , piece_id=2DOVYVWWTRJFW6MUN6ESVEI2AB5CBLN4OPNET7C4OACFIFCP7CQQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615304725 (1h48m39.396238092s) [noise=true , address=104.36.82.7:4912 , piece_id=GS3HZK2674DDXRTAVFFWBMVIO5MIRC4PJVCFKD5MQXDIBLXTXC2A, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615342159 (1h48m39.396275535s) [noise=true , address=104.36.82.4:4801 , piece_id=YVWA5KTVT5QSSOGGOF7XWQKJ2LWK3ECGMPIQJFHOYXC26YWMVVJQ, seg_pos={0 0}, upload=1656]: Done uploading piece. err: ecclient: upload failed (node:12mj4gkN1YNsnfjc9svNEhUo8EAgEmHe7yj4kScHCasB9RwVNwZ, address:104.36.82.4:4801): protocol: rpc: dial tcp 104.36.82.4:4801: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:40:23.615382082 (1h48m39.396315457s) [seg_pos={0 0}, upload=1656]: Done waiting for segment. successful: 7 optimal: 13 errs: <nil> Jul 26 23:40:23.662236247 (1h48m39.443169685s) [upload=1656]: Done uploading segments... Jul 26 23:40:23.662284995 (1h48m39.443218371s) [upload=1656]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 7) Duration: 30.050121088s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 43 out of 60 is failed (threshold=3) [...........X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.028612831s FAIL (2025-07-26 23:40:17 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.028640194s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [.........XX...X...X..XX.X...X..X.......X...X....XX..........] duration: 39.060684706s FAIL (2025-07-26 23:39:55 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 39.060703072s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 42 out of 60 is failed (threshold=3) [............X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.053780697s FAIL (2025-07-26 23:39:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.053796046s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 18 out of 60 is failed (threshold=3) [..................XXX.X....XX.X.X.X.......XXX...XX.....X.XXX] duration: 30.000964872s FAIL (2025-07-26 23:39:36 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001009039s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 38 out of 60 is failed (threshold=3) [......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.034295604s FAIL (2025-07-26 23:39:17 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 5) ======LOGS====== _SegmentBegin Jul 26 23:38:47.421931797 (1h47m3.202865179s) [seg_pos={0 0}, upload=1642]: Begin upload segment... Jul 26 23:38:47.421982422 (1h47m3.202915802s) [seg_pos={0 0}, upload=1642]: Done begin upload segment. Jul 26 23:38:47.422019079 (1h47m3.20295247s) [noise=true , address=104.36.82.7:4910 , piece_id=EGOKV45ZCHIFQ5IPJCYY6WLADSEPDTOCMQ3KQWN3ID3GWLY7SUAQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.422027887 (1h47m3.202961269s) [noise=true , address=104.36.82.10:4077 , piece_id=IR4N6MKUEODMN2YFW3LVW75J7YCKYUTW5E2XQB5ZR7DZS2EJGGYA, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.422501337 (1h47m3.20343472s) [noise=true , address=45.91.116.13:4154 , piece_id=4PCIEOU663PW57PLYKKRXSQG4R3HMAQ4RUHA5ERJ57F6HP6L4UUA, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.422512233 (1h47m3.203445615s) [noise=true , address=104.36.82.16:4295 , piece_id=DBRMWHERYYRWIWRCCROBXXU5VU2AQP7VWWCB6EVTVRWPL5GAF4UA, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.422898572 (1h47m3.203831956s) [noise=true , address=104.36.82.10:4078 , piece_id=CD26AXN5DKNQIFS4OQDZ5L7O3OGJEWNAMGRXUU7QSWSKP7GPCH7A, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.423379839 (1h47m3.204313217s) [noise=true , address=104.36.82.11:4115 , piece_id=ZA5CQGTME7LRJBTJB6LTMFXMUUTFVBPVLAQX7MZJOCZSLD3APSDQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.423781818 (1h47m3.204715198s) [noise=true , address=45.91.116.9:4263 , piece_id=NFUKF5K2S26IU2A23DLFAWJYFR52RLRW64QDMYPE74UMIDGE4FUQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.424278599 (1h47m3.205211978s) [noise=true , address=104.36.82.7:4911 , piece_id=HEPDC2GPDJ4L5DLOUS6VJ3QQN3PQON3ZLDGPRIAC2YYDWRUFJALQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.424676105 (1h47m3.205609485s) [noise=true , address=45.91.116.6:4975 , piece_id=SXD3OD3KSUYOZ6QKDCX7C42CTPXB3WFUUFGJ4PZ4BUYXDMU5IA4Q, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.425058266 (1h47m3.205991648s) [noise=true , address=45.91.116.4:4938 , piece_id=TR3BKO7T5XCOOIRVWQF5ZSCOJQDIAQUNM6E7L3HIZXMBKDYOXPCQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.425439306 (1h47m3.206372674s) [noise=true , address=104.36.82.5:4838 , piece_id=D65E7MQKRI3U3PGSWANWGIED63D47XS4BNOBASXWYKF32I5AXYJQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.425806572 (1h47m3.206739949s) [noise=true , address=104.36.82.15:4257 , piece_id=KMJXIMP4HBOKXCSTOUZE5HA7Z4CAYOPDDNKWXDMLKHVRVFJRV6MA, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.426180477 (1h47m3.207113862s) [noise=true , address=77.90.31.11:4554 , piece_id=G4LS2EIJRIBYZIIX6TX7GYH2RKNKKFT2AOBRMG5XW3F22VP35OYA, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.426580680 (1h47m3.207514049s) [noise=true , address=45.91.116.11:4119 , piece_id=PUWCXXG5QX7CARL7FGNC4QU7OHFQLPODH46ZJ5RRPDU4NHJRNYPA, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.427600434 (1h47m3.208533815s) [noise=true , address=104.36.82.4:4802 , piece_id=OM3LPNKW4SVTZY5AFGGQAQ6DL3OU4AR7DLJZTSEMVX4PVQKBWW5A, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.428074992 (1h47m3.209008366s) [noise=true , address=104.36.82.9:4006 , piece_id=BSG5TOKXZJVZSMRPWXQDSQQBEQVM32BKDW6QZFGWM3SKQ7X3MRUQ, seg_pos={0 0}, upload=1642]: Uploading piece... Jul 26 23:38:47.499282369 (1h47m3.280215738s) [upload=1642]: Next returned nil segment Jul 26 23:38:47.499299554 (1h47m3.280232934s) [upload=1642]: Waiting for error group managing segments... Jul 26 23:38:47.505148623 (1h47m3.286082s) [noise=true , address=45.91.116.6:4975 , piece_id=SXD3OD3KSUYOZ6QKDCX7C42CTPXB3WFUUFGJ4PZ4BUYXDMU5IA4Q, seg_pos={0 0}, upload=1642]: Done uploading piece. err: <nil> Jul 26 23:38:47.505300642 (1h47m3.286234012s) [noise=true , address=45.91.116.13:4154 , piece_id=4PCIEOU663PW57PLYKKRXSQG4R3HMAQ4RUHA5ERJ57F6HP6L4UUA, seg_pos={0 0}, upload=1642]: Done uploading piece. err: <nil> Jul 26 23:38:47.505694113 (1h47m3.286627476s) [noise=true , address=45.91.116.4:4938 , piece_id=TR3BKO7T5XCOOIRVWQF5ZSCOJQDIAQUNM6E7L3HIZXMBKDYOXPCQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: <nil> Jul 26 23:38:47.658447490 (1h47m3.439380859s) [noise=true , address=45.91.116.11:4119 , piece_id=PUWCXXG5QX7CARL7FGNC4QU7OHFQLPODH46ZJ5RRPDU4NHJRNYPA, seg_pos={0 0}, upload=1642]: Done uploading piece. err: <nil> Jul 26 23:38:47.953784843 (1h47m3.734718214s) [noise=true , address=45.91.116.9:4263 , piece_id=NFUKF5K2S26IU2A23DLFAWJYFR52RLRW64QDMYPE74UMIDGE4FUQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: <nil> Jul 26 23:39:17.347362137 (1h47m33.128295532s) [noise=true , address=104.36.82.16:4295 , piece_id=DBRMWHERYYRWIWRCCROBXXU5VU2AQP7VWWCB6EVTVRWPL5GAF4UA, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:1kjzBTGEGEs1qKgbkvvC9bBfPrUa6r6yToz1qhS1pjVqEagauT, address:104.36.82.16:4295): protocol: rpc: dial tcp 104.36.82.16:4295: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347615763 (1h47m33.128549139s) [noise=true , address=104.36.82.10:4077 , piece_id=IR4N6MKUEODMN2YFW3LVW75J7YCKYUTW5E2XQB5ZR7DZS2EJGGYA, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:12kDJuQP8iLUjB1rXpcNX5TtHzsqa5HvYcUD9Ef64HYTnTXcwLk, address:104.36.82.10:4077): protocol: rpc: dial tcp 104.36.82.10:4077: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347692149 (1h47m33.128625575s) [noise=true , address=104.36.82.4:4802 , piece_id=OM3LPNKW4SVTZY5AFGGQAQ6DL3OU4AR7DLJZTSEMVX4PVQKBWW5A, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:1v2RSYwuQZPTLdWGz4QEeEa9STJy1X6HkexD4RYkVjTHGrNcx4, address:104.36.82.4:4802): protocol: write tcp 159.223.143.39:60548->104.36.82.4:4802: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347747760 (1h47m33.12868114s) [noise=true , address=77.90.31.11:4554 , piece_id=G4LS2EIJRIBYZIIX6TX7GYH2RKNKKFT2AOBRMG5XW3F22VP35OYA, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:1eto4dNcMUufd2GixDpEcuXN9ocPRFdnJUPht7vBRdA3BcqRek, address:77.90.31.11:4554): protocol: read tcp 159.223.143.39:59656->77.90.31.11:4554: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347812687 (1h47m33.128746075s) [noise=true , address=104.36.82.11:4115 , piece_id=ZA5CQGTME7LRJBTJB6LTMFXMUUTFVBPVLAQX7MZJOCZSLD3APSDQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: rpc: dial tcp 104.36.82.11:4115: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347869793 (1h47m33.128803169s) [noise=true , address=104.36.82.7:4911 , piece_id=HEPDC2GPDJ4L5DLOUS6VJ3QQN3PQON3ZLDGPRIAC2YYDWRUFJALQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:13rWy7bK7AMZnejVCPwyMs7QrbNcSjWpuVEL8dKX2awG4oJUqu, address:104.36.82.7:4911): protocol: rpc: dial tcp 104.36.82.7:4911: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347928947 (1h47m33.128862311s) [noise=true , address=104.36.82.9:4006 , piece_id=BSG5TOKXZJVZSMRPWXQDSQQBEQVM32BKDW6QZFGWM3SKQ7X3MRUQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:1ig3zGgKZUH8jwXEojp2A6tYGVFCL33PUDkhAbEjS6affBLtmc, address:104.36.82.9:4006): protocol: rpc: dial tcp 104.36.82.9:4006: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347973537 (1h47m33.128906913s) [noise=true , address=104.36.82.5:4838 , piece_id=D65E7MQKRI3U3PGSWANWGIED63D47XS4BNOBASXWYKF32I5AXYJQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:1iSxSmovuEAgNmjE3yCqXKtiRijYbcEcTtUdNxU3iC1ozTxfwL, address:104.36.82.5:4838): protocol: rpc: dial tcp 104.36.82.5:4838: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.347985423 (1h47m33.128918794s) [noise=true , address=104.36.82.10:4078 , piece_id=CD26AXN5DKNQIFS4OQDZ5L7O3OGJEWNAMGRXUU7QSWSKP7GPCH7A, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.348010236 (1h47m33.128943603s) [noise=true , address=104.36.82.15:4257 , piece_id=KMJXIMP4HBOKXCSTOUZE5HA7Z4CAYOPDDNKWXDMLKHVRVFJRV6MA, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:12shbJLfcGs27jH9tFC4L5bQCSNnkatQSXEUyVvDBCJ6oPLLeMK, address:104.36.82.15:4257): protocol: rpc: dial tcp 104.36.82.15:4257: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.348163130 (1h47m33.129096495s) [noise=true , address=104.36.82.7:4910 , piece_id=EGOKV45ZCHIFQ5IPJCYY6WLADSEPDTOCMQ3KQWN3ID3GWLY7SUAQ, seg_pos={0 0}, upload=1642]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: rpc: dial tcp 104.36.82.7:4910: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:17.348240417 (1h47m33.129173795s) [seg_pos={0 0}, upload=1642]: Done waiting for segment. successful: 5 optimal: 13 errs: <nil> Jul 26 23:39:17.380145118 (1h47m33.161078487s) [upload=1642]: Done uploading segments... Jul 26 23:39:17.380169611 (1h47m33.161102993s) [upload=1642]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.034315477s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 13 out of 60 is failed (threshold=3) [..........XX...X...X..XX.X...X..X.......X...X....XX.........] duration: 52.011418003s FAIL (2025-07-26 23:39:16 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 52.011453342s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [XX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.048085333s FAIL (2025-07-26 23:39:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== m_ObjectBegin Jul 26 23:38:10.284561719 (1h46m26.065495097s) [upload=1628]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:38:10.321465178 (1h46m26.102398563s) [seg_pos={0 0}, upload=1628]: Begin upload segment... Jul 26 23:38:10.321530867 (1h46m26.102464232s) [seg_pos={0 0}, upload=1628]: Done begin upload segment. Jul 26 23:38:10.321704631 (1h46m26.102638011s) [noise=true , address=79.127.219.38:4005 , piece_id=QFE26MKRFQ436CZ7WGIXIP2BSKAOG4VOT74PSIXEJ3Z454L5TPIQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.321765633 (1h46m26.102699017s) [noise=true , address=79.127.205.246:4281 , piece_id=433ADJL66C72V6VEJD6MOHVRDCPVSUN6PNKG44QIXHHJ4T7FOETA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.322215816 (1h46m26.103149198s) [noise=true , address=104.36.82.20:4453 , piece_id=BU3NIFK2J3Q3RTUZSKGGTRBOOKQ3KTJLTJR55IEHYB7PVKADMKXQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.323171821 (1h46m26.104105208s) [noise=true , address=77.90.31.19:4849 , piece_id=PATHA4GHVGGHJAQM4UTUIXAUZLPW3ZT3ZDIVK6IAWPIX6WP2VBUA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.323866994 (1h46m26.10480038s) [noise=true , address=79.127.201.215:4350 , piece_id=NYGBEM55XMQDXVQWM33QXCOB536IQLZKJA2I27ANYBUWZJUQ7YXA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.324688778 (1h46m26.105622162s) [noise=true , address=104.36.82.55:4212 , piece_id=ZOSGWHZY2QTNZZV6X4ADRHD57JTXRQJEKDTOIQXVH5K5TGYLSTYA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.325242287 (1h46m26.106175665s) [noise=true , address=79.127.205.233:4643 , piece_id=UTCZTZ4O2LCRKJUMJYOHY57WWPL55ARG36WCGKEVGCT5JR5EAN6Q, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.326044037 (1h46m26.106977422s) [noise=true , address=79.127.219.47:4069 , piece_id=OWQIVWCCMKXX76ZRBQRRXBTJLBAEQEAKPPEYVBLN7QMBDKVVEONA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.326676184 (1h46m26.107609567s) [noise=true , address=104.36.82.46:4513 , piece_id=GIVUVGS6ZLJ4WWMQZM6HVQPHGMTPWYGOZPUZGFCNRYT43VLWESXA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.327510134 (1h46m26.108443522s) [noise=true , address=77.90.31.16:4760 , piece_id=DO5T3BLSIMPKZCTSM23PXWIVIDMCZUCXHLFYSNLXKYRUSRGUEOWQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.327944372 (1h46m26.108877754s) [noise=true , address=79.127.220.98:4422 , piece_id=7HA5PLJLPSTA5BNYAV44IILMURZW7C3Y7F5ZT3NOU5KP52PGO5YA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.328431350 (1h46m26.109364735s) [noise=true , address=79.127.205.234:4616 , piece_id=FYGIXCXURVOZ6EJHBH7DGYQAAS3EGB6DK46SQMWGTJ4X2CZLTH2Q, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.328707695 (1h46m26.109641079s) [noise=true , address=64.49.13.140:4508 , piece_id=XFBJZWZM55CRPX3NGSF25NKTV7ESPPQFLRPWE537IH2R6BO5H7QQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.329331982 (1h46m26.110265367s) [noise=true , address=64.49.13.138:4482 , piece_id=RWLMUBKZZUWF5YXKB3O427MQA7GLM3YX2V56F7QOSYMM4GD2J6KA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.329445281 (1h46m26.110378667s) [noise=true , address=104.36.82.21:4474 , piece_id=YITVHQJZ4IJ6EU2QRMUUEEHEG5WI6PLMBPA5M4NUD4PVB4I6VWFQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.329904045 (1h46m26.110837431s) [noise=true , address=104.36.82.53:4143 , piece_id=NBVFBFTQNWR73UR4JYLVJ36KLVCYK3CI6QDZLIB6G63SXYXGUWYA, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.330629252 (1h46m26.111562635s) [noise=true , address=79.127.205.235:4690 , piece_id=BOL6OJND5T3EKPL6V4AWKX3H6V276L332OXMV7MWFWMWDMU6T2UQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.331398742 (1h46m26.112332129s) [noise=true , address=104.36.82.47:4541 , piece_id=VM6H3II5HCO6A56L5LXCDPZN2T6B5DJWVZEURQJGTTFHVBOLFMGQ, seg_pos={0 0}, upload=1628]: Uploading piece... Jul 26 23:38:10.415942595 (1h46m26.196875972s) [upload=1628]: Next returned nil segment Jul 26 23:38:10.415969697 (1h46m26.196903072s) [upload=1628]: Waiting for error group managing segments... Jul 26 23:38:10.423421258 (1h46m26.20435464s) [noise=true , address=79.127.219.38:4005 , piece_id=QFE26MKRFQ436CZ7WGIXIP2BSKAOG4VOT74PSIXEJ3Z454L5TPIQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:10.450413342 (1h46m26.231346723s) [noise=true , address=77.90.31.16:4760 , piece_id=DO5T3BLSIMPKZCTSM23PXWIVIDMCZUCXHLFYSNLXKYRUSRGUEOWQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:10.507513144 (1h46m26.288446509s) [noise=true , address=79.127.220.98:4422 , piece_id=7HA5PLJLPSTA5BNYAV44IILMURZW7C3Y7F5ZT3NOU5KP52PGO5YA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:10.635830264 (1h46m26.416763631s) [noise=true , address=79.127.219.47:4069 , piece_id=OWQIVWCCMKXX76ZRBQRRXBTJLBAEQEAKPPEYVBLN7QMBDKVVEONA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:10.885921183 (1h46m26.666854544s) [noise=true , address=79.127.205.235:4690 , piece_id=BOL6OJND5T3EKPL6V4AWKX3H6V276L332OXMV7MWFWMWDMU6T2UQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:10.905141234 (1h46m26.686074603s) [noise=true , address=79.127.205.246:4281 , piece_id=433ADJL66C72V6VEJD6MOHVRDCPVSUN6PNKG44QIXHHJ4T7FOETA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:10.945685357 (1h46m26.726618725s) [noise=true , address=79.127.205.233:4643 , piece_id=UTCZTZ4O2LCRKJUMJYOHY57WWPL55ARG36WCGKEVGCT5JR5EAN6Q, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:11.345380090 (1h46m27.126313466s) [noise=true , address=79.127.205.234:4616 , piece_id=FYGIXCXURVOZ6EJHBH7DGYQAAS3EGB6DK46SQMWGTJ4X2CZLTH2Q, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:14.304790376 (1h46m30.085723752s) [noise=true , address=64.49.13.140:4508 , piece_id=XFBJZWZM55CRPX3NGSF25NKTV7ESPPQFLRPWE537IH2R6BO5H7QQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:1diUU2skKzXPjYy3Mu3iDDjnG5W8Z2cxCRMxDCH9cQGg2TdEiR, address:64.49.13.140:4508): protocol: storage node overloaded, request limit: 2000 Jul 26 23:38:14.944937304 (1h46m30.725870692s) [noise=true , address=64.49.13.138:4482 , piece_id=RWLMUBKZZUWF5YXKB3O427MQA7GLM3YX2V56F7QOSYMM4GD2J6KA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:20.402234579 (1h46m36.183167948s) [noise=true , address=77.90.31.19:4849 , piece_id=PATHA4GHVGGHJAQM4UTUIXAUZLPW3ZT3ZDIVK6IAWPIX6WP2VBUA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:38:26.046187863 (1h46m41.827121239s) [noise=true , address=79.127.201.215:4350 , piece_id=NYGBEM55XMQDXVQWM33QXCOB536IQLZKJA2I27ANYBUWZJUQ7YXA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: <nil> Jul 26 23:39:10.252609249 (1h47m26.033542629s) [noise=true , address=104.36.82.53:4143 , piece_id=NBVFBFTQNWR73UR4JYLVJ36KLVCYK3CI6QDZLIB6G63SXYXGUWYA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:12EBnXkxXyf7TYw89gDnoHNpKPyBpNhdT46CipJwjqr6Cg6jxMQ, address:104.36.82.53:4143): protocol: rpc: dial tcp 104.36.82.53:4143: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:10.252723229 (1h47m26.033656605s) [noise=true , address=104.36.82.47:4541 , piece_id=VM6H3II5HCO6A56L5LXCDPZN2T6B5DJWVZEURQJGTTFHVBOLFMGQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:1m1ED9th2QhuoxBKEo3Dync9GpkWjGYNTSBHd57ud2BJoH3HXC, address:104.36.82.47:4541): protocol: rpc: dial tcp 104.36.82.47:4541: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:10.252804093 (1h47m26.033737469s) [noise=true , address=104.36.82.55:4212 , piece_id=ZOSGWHZY2QTNZZV6X4ADRHD57JTXRQJEKDTOIQXVH5K5TGYLSTYA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:1EPo99poctfFTjokBfrawXtUeyraudrXQhXFeyQGe72mjD789y, address:104.36.82.55:4212): protocol: rpc: dial tcp 104.36.82.55:4212: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:10.252882501 (1h47m26.03381588s) [noise=true , address=104.36.82.21:4474 , piece_id=YITVHQJZ4IJ6EU2QRMUUEEHEG5WI6PLMBPA5M4NUD4PVB4I6VWFQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:1rL2DnjSgHP36Md3bCgy5padqTAhv3iQCeURbLMuuuMBxtUNZH, address:104.36.82.21:4474): protocol: rpc: dial tcp 104.36.82.21:4474: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:10.252940420 (1h47m26.033873797s) [noise=true , address=104.36.82.20:4453 , piece_id=BU3NIFK2J3Q3RTUZSKGGTRBOOKQ3KTJLTJR55IEHYB7PVKADMKXQ, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:1qxsiscygr1N83dQS4UE53NJSRcLKsAtuDRCuk7ZXZQ99Yswhi, address:104.36.82.20:4453): protocol: rpc: dial tcp 104.36.82.20:4453: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:10.253019370 (1h47m26.033952748s) [noise=true , address=104.36.82.46:4513 , piece_id=GIVUVGS6ZLJ4WWMQZM6HVQPHGMTPWYGOZPUZGFCNRYT43VLWESXA, seg_pos={0 0}, upload=1628]: Done uploading piece. err: ecclient: upload failed (node:12pyyW8Gdqg6j9YGMtCUBhFnoNXryvPMUy19x815dp2HV3Xqr8h, address:104.36.82.46:4513): protocol: write tcp 159.223.143.39:59284->104.36.82.46:4513: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:39:10.253058304 (1h47m26.033991686s) [seg_pos={0 0}, upload=1628]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:39:10.297369201 (1h47m26.07830257s) [upload=1628]: Done uploading segments... Jul 26 23:39:10.297408551 (1h47m26.078341925s) [upload=1628]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration:* 1m0.048114006s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 41 out of 60 is failed (threshold=3) [.............X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.026301846s FAIL (2025-07-26 23:39:09 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.026330682s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 37 out of 60 is failed (threshold=3) [.......................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.032727023s FAIL (2025-07-26 23:38:47 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 6) ======LOGS====== 26 23:38:17.282381672 (1h46m33.063315054s) [upload=1630]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:38:17.282388081 (1h46m33.063321456s) [upload=1630]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:38:17.325457222 (1h46m33.106390598s) [seg_pos={0 0}, upload=1630]: Begin upload segment... Jul 26 23:38:17.325499025 (1h46m33.10643239s) [seg_pos={0 0}, upload=1630]: Done begin upload segment. Jul 26 23:38:17.325522894 (1h46m33.106456275s) [noise=true , address=77.90.31.16:4734 , piece_id=GHJRLYE6AXZAEQKZ4A3R73GQ3B6DXRGX3VGFD7TQ2LOZRL3V7GVA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.326015952 (1h46m33.106949334s) [noise=true , address=104.36.82.8:4971 , piece_id=7RLCRKTGMDQZ6E5NTU442EUZ2WTXINGMEWJ7Y35QUFHXPAS53C3A, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.326503029 (1h46m33.10743641s) [noise=true , address=45.91.116.6:4975 , piece_id=NJ7MS77HHV3HZS5YLWK35DYMKXRHNYEBOIIITNODINWATDDABFZQ, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.326863575 (1h46m33.10779695s) [noise=true , address=45.91.116.10:4082 , piece_id=MINL34U236G4ZORSPX45HRZFS3PRH4NWNYILCJFE2WOTSUCOJUGA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.327943710 (1h46m33.108877095s) [noise=true , address=104.36.82.7:4912 , piece_id=PPLQXHEUC7RBYND6GPPBMUOOAKDFQZ4K3HKVMNBEEWNEKI2VPUDA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.328763942 (1h46m33.109697321s) [noise=true , address=45.91.116.12:4225 , piece_id=6DAA3NVIJWRYLPUT7GIOWRRFHPTSNJWOHLSKQPMFSWDZB6VBKHMA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.329450902 (1h46m33.110384285s) [noise=true , address=104.36.82.9:4005 , piece_id=J3X3ZUFI7WYT5AOE7R6F7FWS7RWWNDFHRVWJXHYRO2IXJJT4RBNA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.329827291 (1h46m33.110760669s) [noise=true , address=104.36.82.12:4150 , piece_id=SLRKLGRC3N5DQ56ICKET243DWDXYOS6KKZPQLQDHXOZEIJ3UONUA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.329914440 (1h46m33.110847824s) [noise=true , address=45.91.116.9:4263 , piece_id=NR7ROCNVY76S5TRN2IP3QUVZAP7RORIII4ATL2QWU7VN4EKDG3CQ, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.330382271 (1h46m33.111315649s) [noise=true , address=104.36.82.4:4804 , piece_id=CXGM45OCDQ23LJ3UXOMIR46V5QONEDF7E4DSGKHJLP3SEZ23GMDQ, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.330842660 (1h46m33.111776042s) [noise=true , address=45.91.116.4:4937 , piece_id=TARXCM4JN5PYNBULYPBZWE53Y4WPV2ZIWIN4SCSBEI6VCO55S5FA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.331249859 (1h46m33.112183237s) [noise=true , address=104.36.82.2:4729 , piece_id=LRX7X6MZISWVBGG6FSLGBSAJXQDWYF6QXNBPR3SW3AQCNNO4DL2Q, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.331700245 (1h46m33.112633631s) [noise=true , address=104.36.82.10:4078 , piece_id=UNO7ZD6BN67INUQFRZEMLVIQVFLJMZBBJEQJISDZYWK6RUQVVLGA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.332193010 (1h46m33.113126396s) [noise=true , address=104.36.82.12:4151 , piece_id=4JSTZ7WBRM3ZMO3MHMVJDIZZPKKNGN4LBVG24GGOI6IA7NFU7MMA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.332877279 (1h46m33.113810664s) [noise=true , address=104.36.82.6:4875 , piece_id=OGMKDBNZUM5C456KDEMQBL233ZG4STKKG2UPWY6YELTFXSCRCEUA, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.333406079 (1h46m33.114339466s) [noise=true , address=104.36.82.11:4115 , piece_id=24X2WT64KHLE3FJVGI5CKXQMUWDVEDAWA4PHU3EFNRLH7XKQ5CUQ, seg_pos={0 0}, upload=1630]: Uploading piece... Jul 26 23:38:17.415164740 (1h46m33.19609812s) [upload=1630]: Next returned nil segment Jul 26 23:38:17.415188443 (1h46m33.19612183s) [upload=1630]: Waiting for error group managing segments... Jul 26 23:38:17.419552774 (1h46m33.200486142s) [noise=true , address=45.91.116.4:4937 , piece_id=TARXCM4JN5PYNBULYPBZWE53Y4WPV2ZIWIN4SCSBEI6VCO55S5FA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: <nil> Jul 26 23:38:17.420179959 (1h46m33.201113325s) [noise=true , address=45.91.116.6:4975 , piece_id=NJ7MS77HHV3HZS5YLWK35DYMKXRHNYEBOIIITNODINWATDDABFZQ, seg_pos={0 0}, upload=1630]: Done uploading piece. err: <nil> Jul 26 23:38:17.420724133 (1h46m33.201657498s) [noise=true , address=45.91.116.10:4082 , piece_id=MINL34U236G4ZORSPX45HRZFS3PRH4NWNYILCJFE2WOTSUCOJUGA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: <nil> Jul 26 23:38:17.421161211 (1h46m33.202094586s) [noise=true , address=45.91.116.12:4225 , piece_id=6DAA3NVIJWRYLPUT7GIOWRRFHPTSNJWOHLSKQPMFSWDZB6VBKHMA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: <nil> Jul 26 23:38:17.421636875 (1h46m33.202570244s) [noise=true , address=77.90.31.16:4734 , piece_id=GHJRLYE6AXZAEQKZ4A3R73GQ3B6DXRGX3VGFD7TQ2LOZRL3V7GVA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: <nil> Jul 26 23:38:17.483748356 (1h46m33.264681732s) [noise=true , address=45.91.116.9:4263 , piece_id=NR7ROCNVY76S5TRN2IP3QUVZAP7RORIII4ATL2QWU7VN4EKDG3CQ, seg_pos={0 0}, upload=1630]: Done uploading piece. err: <nil> Jul 26 23:38:47.247103928 (1h47m3.028037302s) [noise=true , address=104.36.82.12:4150 , piece_id=SLRKLGRC3N5DQ56ICKET243DWDXYOS6KKZPQLQDHXOZEIJ3UONUA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: rpc: dial tcp 104.36.82.12:4150: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247174663 (1h47m3.028108028s) [noise=true , address=104.36.82.4:4804 , piece_id=CXGM45OCDQ23LJ3UXOMIR46V5QONEDF7E4DSGKHJLP3SEZ23GMDQ, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247208890 (1h47m3.02814228s) [noise=true , address=104.36.82.10:4078 , piece_id=UNO7ZD6BN67INUQFRZEMLVIQVFLJMZBBJEQJISDZYWK6RUQVVLGA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: rpc: dial tcp 104.36.82.10:4078: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247224212 (1h47m3.028157578s) [noise=true , address=104.36.82.2:4729 , piece_id=LRX7X6MZISWVBGG6FSLGBSAJXQDWYF6QXNBPR3SW3AQCNNO4DL2Q, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:1BKzosFpHWqYNaAck7XPxK263dJ4bPZ6hVSYye65CJa1uHRmoR, address:104.36.82.2:4729): protocol: rpc: dial tcp 104.36.82.2:4729: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247274888 (1h47m3.028208253s) [noise=true , address=104.36.82.8:4971 , piece_id=7RLCRKTGMDQZ6E5NTU442EUZ2WTXINGMEWJ7Y35QUFHXPAS53C3A, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:12Ut4J9xtQTDXFFmCSv9rk6sBSjNpEdnyxo6cZWMXcpAvuQtebo, address:104.36.82.8:4971): protocol: rpc: dial tcp 104.36.82.8:4971: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247297158 (1h47m3.028230534s) [noise=true , address=104.36.82.9:4005 , piece_id=J3X3ZUFI7WYT5AOE7R6F7FWS7RWWNDFHRVWJXHYRO2IXJJT4RBNA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:12YnzPGyppEZ97NvHxEQiSK18T1bctGqPzdDrXzLGeCSRUBj2WJ, address:104.36.82.9:4005): protocol: rpc: dial tcp 104.36.82.9:4005: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247325516 (1h47m3.028258881s) [noise=true , address=104.36.82.7:4912 , piece_id=PPLQXHEUC7RBYND6GPPBMUOOAKDFQZ4K3HKVMNBEEWNEKI2VPUDA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:42306->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247363193 (1h47m3.028296558s) [noise=true , address=104.36.82.6:4875 , piece_id=OGMKDBNZUM5C456KDEMQBL233ZG4STKKG2UPWY6YELTFXSCRCEUA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:1rxmCJKvyoD8UiguHRdctfRTQ1Fiu9x29M5Xbf69ihF8uBG6m2, address:104.36.82.6:4875): protocol: rpc: dial tcp 104.36.82.6:4875: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247388224 (1h47m3.028321593s) [noise=true , address=104.36.82.12:4151 , piece_id=4JSTZ7WBRM3ZMO3MHMVJDIZZPKKNGN4LBVG24GGOI6IA7NFU7MMA, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:1w6pwZMdeQKjt92qx6WbTWU5VPYJto5gS87f6iKvCBZH6LfyeB, address:104.36.82.12:4151): protocol: rpc: dial tcp 104.36.82.12:4151: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247425257 (1h47m3.02835863s) [noise=true , address=104.36.82.11:4115 , piece_id=24X2WT64KHLE3FJVGI5CKXQMUWDVEDAWA4PHU3EFNRLH7XKQ5CUQ, seg_pos={0 0}, upload=1630]: Done uploading piece. err: ecclient: upload failed (node:12DKJTyYZ5om3b3NgCn2HXBwGbN3aoi2fk6gb9H5gAbVbHFaeoR, address:104.36.82.11:4115): protocol: write tcp 159.223.143.39:40042->104.36.82.11:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:47.247465754 (1h47m3.028399139s) [seg_pos={0 0}, upload=1630]: Done waiting for segment. successful: 6 optimal: 13 errs: <nil> Jul 26 23:38:47.277303759 (1h47m3.05823714s) [upload=1630]: Done uploading segments... Jul 26 23:38:47.277332606 (1h47m3.058265987s) [upload=1630]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.03275199s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 40 out of 60 is failed (threshold=3) [..............X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.035724491s FAIL (2025-07-26 23:38:30 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 6) Duration: 30.035751986s core uplink us1 vivint
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 17 out of 60 is failed (threshold=3) [...................XXX.X....XX.X.X.X.......XXX...XX.....X.XX] duration: 30.000537323s FAIL (2025-07-26 23:38:26 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000551226s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 50 out of 60 is failed (threshold=3) [XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.209847972s FAIL (2025-07-26 23:38:05 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 11) ======LOGS====== batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:37:05.096773169 (1h45m20.877706558s) [seg_pos={0 0}, upload=1612]: Begin upload segment... Jul 26 23:37:05.096857219 (1h45m20.877790586s) [seg_pos={0 0}, upload=1612]: Done begin upload segment. Jul 26 23:37:05.096908307 (1h45m20.877841696s) [noise=true , address=79.127.220.101:4048 , piece_id=BNVOMJPTRGIBZCZLGMTZAZ65E4V75ZYH57VUMRWPSBXTOA4J3ALQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.097588314 (1h45m20.878521704s) [noise=true , address=79.127.201.215:4344 , piece_id=OO7W7TLFHLYV5KYDOU4BTKW4IFFKZ7O3GI7OPFBL6ESMZ4WJHYDQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.098136937 (1h45m20.879070327s) [noise=true , address=79.127.205.245:4232 , piece_id=SOPDQGL2JIAUTDZLMLC3PR4U5B3QZA73537GYCO47K4MZMBOMPAQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.098661917 (1h45m20.879595294s) [noise=true , address=79.127.219.38:4010 , piece_id=ZW4RFLLDM5P37SC4AK3ZXRIZW3GDXUZB4FLKYHCKW43RHM37WOOQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.099218376 (1h45m20.880151765s) [noise=true , address=104.36.82.46:4489 , piece_id=6REVHFFE7A5Z6U3T5UGIIZQU6NKA6E6GMBGUZTGFDNOBO557MTTQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.099720240 (1h45m20.880653634s) [noise=true , address=104.36.82.14:4255 , piece_id=25QTE3D4J2FIS73QEMHHBTJA5RGJGUUIGTKP47C4M6ZXU77ZZ5RQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.114224946 (1h45m20.895158571s) [noise=true , address=104.36.82.47:4537 , piece_id=GDEAWPJQQAGMDCPNNAS22OCVX4JUMLRI5JSL6GURTRJLACOPZN6A, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.114755110 (1h45m20.895688499s) [noise=true , address=79.127.205.234:4613 , piece_id=EV4IC5KS4G7ZKTRXUPNCDUW4PPJB4U4H75KPQLNJ73CXPNLGUYBQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.115253358 (1h45m20.896186739s) [noise=true , address=45.91.116.14:4206 , piece_id=TOFCQ2KD7AB4ANQLNXE4KTME2CWGQXXYZKOM5CVZYTZC5NA6IJKA, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.115620862 (1h45m20.896554231s) [noise=true , address=45.91.116.5:4306 , piece_id=UKT6IJ57R6SNKAI5ZBJC3WRRTULAFLTW3AG6WH5BVOA25XRHQP4Q, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.116183317 (1h45m20.897116706s) [noise=true , address=79.127.205.246:4288 , piece_id=KN2CT2T6SYXEB5IVJKUFVCPB7ESHUPHSZE5NL457YIRIRG3S5XFQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.116667914 (1h45m20.897601299s) [noise=true , address=104.36.82.56:4253 , piece_id=WGOYQBXJPUMYQXQW5ATXNZ6K4FS55EIFCYWDB3YNQ2B6UHCHJXDA, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.117658564 (1h45m20.89859195s) [noise=true , address=156.146.43.225:4434 , piece_id=PRVUI6LFEJJVEQNU4PTEZ6LIVKOE6E6XBBDSZHJJOYNKOPIKR4JA, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.118129485 (1h45m20.899062874s) [noise=true , address=45.91.116.12:4230 , piece_id=IHLW7BEYQOH2J3WZOAA33B7D7D2CYS5VKWFPZ4A3OFOQIY5EQV3A, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.118583519 (1h45m20.899516887s) [noise=true , address=79.127.205.233:4639 , piece_id=V6QFNCYNVTYXYY72RT2RSFCSYAC65JNNDNQTUSKGDGSRF4SGSWEQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.119004357 (1h45m20.899937746s) [noise=true , address=104.36.82.55:4195 , piece_id=HEUPCCF7MQ2CUFN2BAUXSUQLDQTFN3NF6VI4GQOYL6FXYNWF4MMQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.119365902 (1h45m20.900299289s) [noise=true , address=45.91.116.14:4194 , piece_id=77YICZE7GUHW6RHRTIHRPM36BMYWSVRX5R34VBYBFNOLD5RVS6LQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.119798901 (1h45m20.900732285s) [noise=true , address=104.36.82.52:4115 , piece_id=HQFSWNP2CPBSATF6XU3E3UINJWV627VBA75JT33V7LAMDGKJWXNQ, seg_pos={0 0}, upload=1612]: Uploading piece... Jul 26 23:37:05.173071095 (1h45m20.95400448s) [upload=1612]: Next returned nil segment Jul 26 23:37:05.173093064 (1h45m20.954026449s) [upload=1612]: Waiting for error group managing segments... Jul 26 23:37:05.177523410 (1h45m20.958456788s) [noise=true , address=79.127.219.38:4010 , piece_id=ZW4RFLLDM5P37SC4AK3ZXRIZW3GDXUZB4FLKYHCKW43RHM37WOOQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:05.215155675 (1h45m20.996089056s) [noise=true , address=79.127.220.101:4048 , piece_id=BNVOMJPTRGIBZCZLGMTZAZ65E4V75ZYH57VUMRWPSBXTOA4J3ALQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: ecclient: upload failed (node:1csdxFUHCWKuGfrZnVfpRJ3R6DYCzxx3voY1UR5kWwSXaxZaPg, address:79.127.220.101:4048): protocol: storage node overloaded, request limit: 2000 Jul 26 23:37:05.628395718 (1h45m21.40932909s) [noise=true , address=45.91.116.14:4194 , piece_id=77YICZE7GUHW6RHRTIHRPM36BMYWSVRX5R34VBYBFNOLD5RVS6LQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:05.712600929 (1h45m21.493534298s) [noise=true , address=79.127.205.233:4639 , piece_id=V6QFNCYNVTYXYY72RT2RSFCSYAC65JNNDNQTUSKGDGSRF4SGSWEQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:05.733477744 (1h45m21.514411113s) [noise=true , address=79.127.205.245:4232 , piece_id=SOPDQGL2JIAUTDZLMLC3PR4U5B3QZA73537GYCO47K4MZMBOMPAQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:05.787828945 (1h45m21.568762315s) [noise=true , address=79.127.205.246:4288 , piece_id=KN2CT2T6SYXEB5IVJKUFVCPB7ESHUPHSZE5NL457YIRIRG3S5XFQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:05.844604883 (1h45m21.62553826s) [noise=true , address=79.127.205.234:4613 , piece_id=EV4IC5KS4G7ZKTRXUPNCDUW4PPJB4U4H75KPQLNJ73CXPNLGUYBQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:05.929656818 (1h45m21.710590181s) [noise=true , address=45.91.116.5:4306 , piece_id=UKT6IJ57R6SNKAI5ZBJC3WRRTULAFLTW3AG6WH5BVOA25XRHQP4Q, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:06.351490290 (1h45m22.132423658s) [noise=true , address=79.127.201.215:4344 , piece_id=OO7W7TLFHLYV5KYDOU4BTKW4IFFKZ7O3GI7OPFBL6ESMZ4WJHYDQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:08.511765295 (1h45m24.292698674s) [noise=true , address=156.146.43.225:4434 , piece_id=PRVUI6LFEJJVEQNU4PTEZ6LIVKOE6E6XBBDSZHJJOYNKOPIKR4JA, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:09.818107843 (1h45m25.599041227s) [noise=true , address=45.91.116.14:4206 , piece_id=TOFCQ2KD7AB4ANQLNXE4KTME2CWGQXXYZKOM5CVZYTZC5NA6IJKA, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:37:12.545384021 (1h45m28.326317398s) [noise=true , address=45.91.116.12:4230 , piece_id=IHLW7BEYQOH2J3WZOAA33B7D7D2CYS5VKWFPZ4A3OFOQIY5EQV3A, seg_pos={0 0}, upload=1612]: Done uploading piece. err: <nil> Jul 26 23:38:04.836492282 (1h46m20.617425667s) [noise=true , address=104.36.82.46:4489 , piece_id=6REVHFFE7A5Z6U3T5UGIIZQU6NKA6E6GMBGUZTGFDNOBO557MTTQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: ecclient: upload failed (node:12YbmXQeh182PYH1AHxvGcw3zW3uiuDsbYPomH3HAPNtjbJSpPw, address:104.36.82.46:4489): protocol: rpc: dial tcp 104.36.82.46:4489: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:04.836547165 (1h46m20.617480559s) [noise=true , address=104.36.82.55:4195 , piece_id=HEUPCCF7MQ2CUFN2BAUXSUQLDQTFN3NF6VI4GQOYL6FXYNWF4MMQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: ecclient: upload failed (node:12cizR87omcPERJsRZt64xHfeQyw6BUpBsJPB2txS4LjZR9StMa, address:104.36.82.55:4195): protocol: rpc: dial tcp 104.36.82.55:4195: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:04.836612842 (1h46m20.617546221s) [noise=true , address=104.36.82.14:4255 , piece_id=25QTE3D4J2FIS73QEMHHBTJA5RGJGUUIGTKP47C4M6ZXU77ZZ5RQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: ecclient: upload failed (node:12fTNKdc6AZbkfsKRFZwTwV6wPz9j9ZD9t2eNkHBJNLMNcGh9gR, address:104.36.82.14:4255): protocol: write tcp 159.223.143.39:57756->104.36.82.14:4255: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:04.836628262 (1h46m20.617561627s) [noise=true , address=104.36.82.47:4537 , piece_id=GDEAWPJQQAGMDCPNNAS22OCVX4JUMLRI5JSL6GURTRJLACOPZN6A, seg_pos={0 0}, upload=1612]: Done uploading piece. err: ecclient: upload failed (node:12LTTCDNikuwf94KeKsQGFSu6ABXVYt1ombRsEyrZN5JC8d4Eot, address:104.36.82.47:4537): protocol: rpc: dial tcp 104.36.82.47:4537: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:04.836685559 (1h46m20.61761905s) [noise=true , address=104.36.82.56:4253 , piece_id=WGOYQBXJPUMYQXQW5ATXNZ6K4FS55EIFCYWDB3YNQ2B6UHCHJXDA, seg_pos={0 0}, upload=1612]: Done uploading piece. err: ecclient: upload failed (node:1Pz6ZrFzAdLySCxtFhbotfPLiA4nSmvhp5uaRAJQgSc7DUgmap, address:104.36.82.56:4253): protocol: write tcp 159.223.143.39:35020->104.36.82.56:4253: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:04.836698859 (1h46m20.617632238s) [noise=true , address=104.36.82.52:4115 , piece_id=HQFSWNP2CPBSATF6XU3E3UINJWV627VBA75JT33V7LAMDGKJWXNQ, seg_pos={0 0}, upload=1612]: Done uploading piece. err: ecclient: upload failed (node:1ikQ2b343QztY3vBFQ3cmEwEEFszpB2TPMtstufHdFfMEkJkBV, address:104.36.82.52:4115): protocol: write tcp 159.223.143.39:57942->104.36.82.52:4115: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:38:04.836722705 (1h46m20.617656088s) [seg_pos={0 0}, upload=1612]: Done waiting for segment. successful: 11 optimal: 14 errs: <nil> Jul 26 23:38:05.043294997 (1h46m20.824228366s) [upload=1612]: Done uploading segments... Jul 26 23:38:05.043324151 (1h46m20.824257519s) [upload=1612]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 11) Duration:* 1m0.209872524s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 16 out of 60 is failed (threshold=3) [....................XXX.X....XX.X.X.X.......XXX...XX.....X.X] duration: 30.00074243s FAIL (2025-07-26 23:37:56 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.000839161s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 39 out of 60 is failed (threshold=3) [...............X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.033373309s FAIL (2025-07-26 23:37:39 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.033414766s core uplink us1 vivint
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [X..........XX...X...X..XX.X...X..X.......X...X....XX........] duration: 7.687785762s FAIL (2025-07-26 23:37:37 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 7.687814176s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 36 out of 60 is failed (threshold=3) [........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.026265514s FAIL (2025-07-26 23:37:35 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 11) ======LOGS====== XKRW2LU24SA, seg_pos={0 0}, upload=1595]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:56724->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:23.914569057 (1h44m39.69550244s) [seg_pos={0 0}, upload=1595]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:36:23.942316640 (1h44m39.723250007s) [upload=1595]: Done uploading segments... Jul 26 23:36:23.942360746 (1h44m39.723294114s) [upload=1595]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 26 23:37:05.883283406 (1h45m21.664216789s) [upload=1614]: Starting upload Jul 26 23:37:05.883396987 (1h45m21.664330374s) [upload=1614]: Done starting upload Jul 26 23:37:05.883550745 (1h45m21.664484121s) [upload=1614]: Uploading segments... Jul 26 23:37:05.885260248 (1h45m21.666193626s) [upload=1614]: Got next segment. Inline: false Jul 26 23:37:05.885374047 (1h45m21.666307429s) [upload=1614]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:37:05.885381923 (1h45m21.666315306s) [upload=1614]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:37:05.916570243 (1h45m21.697503656s) [seg_pos={0 0}, upload=1614]: Begin upload segment... Jul 26 23:37:05.916642029 (1h45m21.697575415s) [seg_pos={0 0}, upload=1614]: Done begin upload segment. Jul 26 23:37:05.916791900 (1h45m21.697725286s) [noise=true , address=104.36.82.5:4837 , piece_id=3ZGRR77BNNACN36CM4LJMEVA7DWAKPBCYZAR4XLOCLKHHXUYT5DA, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.916848880 (1h45m21.697782271s) [noise=true , address=45.91.116.4:4938 , piece_id=PKRA7ZRCFLYXG6AKWBNQ65SUP5WCIAC7RM44VVDKMZNLC6T65SXQ, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.917352874 (1h45m21.69828626s) [noise=true , address=77.90.31.16:4733 , piece_id=EWDITDG3E4XDBVIFNKEMLGELY3NSKUFMEIE3RLFN7T26JNWNDSFA, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.918915294 (1h45m21.699848675s) [noise=true , address=45.91.116.13:4154 , piece_id=GZKEJNBYNPOMK76SW4KMLS6EFIFL3QV2VSEX2LPPFOAQKTDTZS6Q, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.919768361 (1h45m21.70070175s) [noise=true , address=45.91.116.8:4048 , piece_id=7YZCLJRYL7PKTVEQNGYJK3JEGDUSMYWMPAVQ2GZBKPTZIHPM7ZTA, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.920640421 (1h45m21.701573807s) [noise=true , address=77.90.31.13:4628 , piece_id=CGP4ZJZVXYMAQXRVO2SX3EVRLK7YDSSE64TCUPYTDKA7M3LUCOTA, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.923212326 (1h45m21.704145715s) [noise=true , address=104.36.82.12:4149 , piece_id=LXJWCOI45R6XRQSG63SXPDYIJGWR3FZXVRCRVDGYZ25CHLBJ3FPA, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.924383087 (1h45m21.70531647s) [noise=true , address=77.90.31.18:4806 , piece_id=SIIJCEA2EU7KRYV3Y4NSHKCDQBY2P7GTJKDVM3ARWJKK5Q5U5TCA, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.925189703 (1h45m21.706123094s) [noise=true , address=77.90.31.19:4843 , piece_id=B5NR7IJUVLK4OIROD63FVMUH764RIM4C4UL5H5RAEMKUNRKQK62Q, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.926252181 (1h45m21.70718557s) [noise=true , address=104.36.82.7:4912 , piece_id=D5VCAUEGBTZCY75OA225RQGNH2B2Q23JFI5363G6SB7F4HUX347Q, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.927080241 (1h45m21.708013631s) [noise=true , address=77.90.31.10:4518 , piece_id=Y36RWUQBORNLUQKMEDREYKK6ED6PV4TTRXZCHBGV2PW2MESWPBIA, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.928100694 (1h45m21.709034124s) [noise=true , address=45.91.116.7:4009 , piece_id=YAPWDQTQA4AL6VAFUPTKAJUCM7SBXF3JJND6HMBZBZ5BCIDFJ5VQ, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.929246960 (1h45m21.710180343s) [noise=true , address=45.91.116.5:4299 , piece_id=3HHZWA4NMPPFXAIANZ6CE26Z2XXL3AHYHLQZ7L4PC526WXAS6S4Q, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.929705326 (1h45m21.710638712s) [noise=true , address=104.36.82.6:4873 , piece_id=MEQ3OR2ZVQ6RUH35J5QXW3SME7VFQVZH7EGKPKOPB77CUFB2YIIA, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.930074181 (1h45m21.711007569s) [noise=true , address=104.36.82.4:4804 , piece_id=53JXL5ECNNK2CYGWVV55UOBJRZNU6NNFXYFSPH2VC4E4DO3QA24A, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:05.930456851 (1h45m21.711390239s) [noise=true , address=45.91.116.12:4227 , piece_id=WJKUAZQKDDGTYYPXWGNBWGCKJUM7OD3ZMPAP6BILZR7OI5ZHQ44A, seg_pos={0 0}, upload=1614]: Uploading piece... Jul 26 23:37:06.054626560 (1h45m21.835559922s) [upload=1614]: Next returned nil segment Jul 26 23:37:06.054646894 (1h45m21.835580258s) [upload=1614]: Waiting for error group managing segments... Jul 26 23:37:06.060866261 (1h45m21.841799625s) [noise=true , address=45.91.116.4:4938 , piece_id=PKRA7ZRCFLYXG6AKWBNQ65SUP5WCIAC7RM44VVDKMZNLC6T65SXQ, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:06.061930323 (1h45m21.842863686s) [noise=true , address=45.91.116.7:4009 , piece_id=YAPWDQTQA4AL6VAFUPTKAJUCM7SBXF3JJND6HMBZBZ5BCIDFJ5VQ, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:06.064180345 (1h45m21.845113714s) [noise=true , address=77.90.31.18:4806 , piece_id=SIIJCEA2EU7KRYV3Y4NSHKCDQBY2P7GTJKDVM3ARWJKK5Q5U5TCA, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:06.065098049 (1h45m21.846031419s) [noise=true , address=45.91.116.8:4048 , piece_id=7YZCLJRYL7PKTVEQNGYJK3JEGDUSMYWMPAVQ2GZBKPTZIHPM7ZTA, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:06.066996167 (1h45m21.847929538s) [noise=true , address=45.91.116.13:4154 , piece_id=GZKEJNBYNPOMK76SW4KMLS6EFIFL3QV2VSEX2LPPFOAQKTDTZS6Q, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:06.071454152 (1h45m21.852387528s) [noise=true , address=45.91.116.12:4227 , piece_id=WJKUAZQKDDGTYYPXWGNBWGCKJUM7OD3ZMPAP6BILZR7OI5ZHQ44A, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:06.112162673 (1h45m21.893096052s) [noise=true , address=77.90.31.16:4733 , piece_id=EWDITDG3E4XDBVIFNKEMLGELY3NSKUFMEIE3RLFN7T26JNWNDSFA, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:06.138091162 (1h45m21.919024535s) [noise=true , address=45.91.116.5:4299 , piece_id=3HHZWA4NMPPFXAIANZ6CE26Z2XXL3AHYHLQZ7L4PC526WXAS6S4Q, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:07.464697484 (1h45m23.245630855s) [noise=true , address=77.90.31.19:4843 , piece_id=B5NR7IJUVLK4OIROD63FVMUH764RIM4C4UL5H5RAEMKUNRKQK62Q, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:07.490168140 (1h45m23.271101516s) [noise=true , address=77.90.31.13:4628 , piece_id=CGP4ZJZVXYMAQXRVO2SX3EVRLK7YDSSE64TCUPYTDKA7M3LUCOTA, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:07.518510416 (1h45m23.299443795s) [noise=true , address=77.90.31.10:4518 , piece_id=Y36RWUQBORNLUQKMEDREYKK6ED6PV4TTRXZCHBGV2PW2MESWPBIA, seg_pos={0 0}, upload=1614]: Done uploading piece. err: <nil> Jul 26 23:37:35.853485599 (1h45m51.634418966s) [noise=true , address=104.36.82.4:4804 , piece_id=53JXL5ECNNK2CYGWVV55UOBJRZNU6NNFXYFSPH2VC4E4DO3QA24A, seg_pos={0 0}, upload=1614]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: write tcp 159.223.143.39:47272->104.36.82.4:4804: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:35.853562766 (1h45m51.634496142s) [noise=true , address=104.36.82.5:4837 , piece_id=3ZGRR77BNNACN36CM4LJMEVA7DWAKPBCYZAR4XLOCLKHHXUYT5DA, seg_pos={0 0}, upload=1614]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:35.853618022 (1h45m51.634551388s) [noise=true , address=104.36.82.6:4873 , piece_id=MEQ3OR2ZVQ6RUH35J5QXW3SME7VFQVZH7EGKPKOPB77CUFB2YIIA, seg_pos={0 0}, upload=1614]: Done uploading piece. err: ecclient: upload failed (node:12XMLfJdbmhqnSjxRQ6DJwJowhxiqYzvi34JArdHpLzr8r2xnQ3, address:104.36.82.6:4873): protocol: write tcp 159.223.143.39:47036->104.36.82.6:4873: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:35.853669494 (1h45m51.63460286s) [noise=true , address=104.36.82.12:4149 , piece_id=LXJWCOI45R6XRQSG63SXPDYIJGWR3FZXVRCRVDGYZ25CHLBJ3FPA, seg_pos={0 0}, upload=1614]: Done uploading piece. err: ecclient: upload failed (node:12qexRLJM6TY935UaEWsj1X9ViyBvk2BM1XGjnq8kapnXXXmgbm, address:104.36.82.12:4149): protocol: write tcp 159.223.143.39:51778->104.36.82.12:4149: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:35.853721476 (1h45m51.63465485s) [noise=true , address=104.36.82.7:4912 , piece_id=D5VCAUEGBTZCY75OA225RQGNH2B2Q23JFI5363G6SB7F4HUX347Q, seg_pos={0 0}, upload=1614]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: rpc: dial tcp 104.36.82.7:4912: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:35.853743014 (1h45m51.634676391s) [seg_pos={0 0}, upload=1614]: Done waiting for segment. successful: 11 optimal: 13 errs: <nil> Jul 26 23:37:35.877769449 (1h45m51.658702812s) [upload=1614]: Done uploading segments... Jul 26 23:37:35.877802051 (1h45m51.658735432s) [upload=1614]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 11) Duration: 30.026293224s core uplink us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.035030859s FAIL (2025-07-26 23:37:04 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 12) ======LOGS====== 1h44m20.573930382s) [upload=1598]: Uploading segments... Jul 26 23:36:04.793326392 (1h44m20.574259777s) [upload=1598]: Got next segment. Inline: false Jul 26 23:36:04.793422468 (1h44m20.574355851s) [upload=1598]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:36:04.793431673 (1h44m20.574365059s) [upload=1598]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:36:04.839132394 (1h44m20.62006578s) [seg_pos={0 0}, upload=1598]: Begin upload segment... Jul 26 23:36:04.839230235 (1h44m20.620163625s) [noise=true , address=104.36.82.54:4187 , piece_id=BIWDUNNMQZGGWA7P3HOHLVEWC3LY7UUWORMY6LIOK7UN55BA55BQ, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.839744212 (1h44m20.620677596s) [noise=true , address=79.127.219.51:4196 , piece_id=U4NDAUEAXGWNBALDD32JBGNQLZQPE4T6IILT7JPWUE6WWNMYFBCQ, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.840339521 (1h44m20.621272911s) [noise=true , address=79.127.205.245:4237 , piece_id=WGJX7TMGFWAS3ECOHZXNTNUKDXGDCYQX4RFJU2LTHHC3OE335DGQ, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.840697723 (1h44m20.621631112s) [noise=true , address=77.90.31.13:4658 , piece_id=DBR7S55I4MPSYYU6RZ53YPY7LRGOC7NLGJTPJN5WDKKSDFKK7XVQ, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.841007619 (1h44m20.621940997s) [noise=true , address=45.91.116.14:4194 , piece_id=ASNDK6UDUBYSWHRCAXXEZNDOOWHAVGTY4RLADLSBKOK6FQW7GL3Q, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.845339432 (1h44m20.626272821s) [seg_pos={0 0}, upload=1598]: Done begin upload segment. Jul 26 23:36:04.845652152 (1h44m20.626585541s) [noise=true , address=79.127.201.215:4358 , piece_id=A7C4IOLSJSOESPQH272ZD7Z4L4RRRTD73NFUPK427K35BXZQT4YQ, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.846510648 (1h44m20.627444036s) [noise=true , address=64.49.13.140:4508 , piece_id=NUTBHS4EYXX4CTBYXQO2FVICUEO4ZONBZORHKR5M7GYBMREAINKQ, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.851906006 (1h44m20.632839391s) [noise=true , address=79.127.220.98:4421 , piece_id=FA56MS4AYJR7BDEJYF6FRIY6EPRA3IBUXKBBN6BNW6GVTIFB4JWA, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.852726707 (1h44m20.633660093s) [noise=true , address=79.127.205.235:4689 , piece_id=7OEOLJPAZX6RTIRRJW3HNBGG3CC6UCORPOGWSCBUZGM6AHA7BNXA, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.854274177 (1h44m20.635207559s) [noise=true , address=77.90.31.18:4833 , piece_id=I2QO2PXWEEHCXK4AUSJRS74DI52ACO6SJZKN73TIX3DQYYQNN6PA, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.859614926 (1h44m20.640548341s) [noise=true , address=79.127.219.38:4010 , piece_id=TXBGG5A5OU7M4DAGIONF3JD35M2GA4CKCOWY2LE6I4TT6DXGAE4A, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.860440553 (1h44m20.641373924s) [noise=true , address=104.36.82.46:4481 , piece_id=2RYU5YI4BCG5FUIDMQLOI4OAB3SJ2OPZ7OUTDISNWR3GXOW2GU5A, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.870288756 (1h44m20.651222136s) [noise=true , address=104.36.82.55:4224 , piece_id=ERBTARIXECO7WA5XV3UE7G7QMAKK6PVSZAX75SB73ZH6URX6IOYA, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.870666345 (1h44m20.65159973s) [noise=true , address=79.127.205.246:4281 , piece_id=PDXNFEHEAI3TSG3Z5EXHZSK4ZQ672ISPX73WSD7AP4SM55NJFPRQ, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.875966232 (1h44m20.656899625s) [noise=true , address=104.36.82.47:4537 , piece_id=FJZIQOODCFHS7FIVA4ZKTE2HRMPGTP5MY257VV4CYJW4Y5JV65NQ, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.876709751 (1h44m20.65764314s) [noise=true , address=79.127.205.233:4646 , piece_id=QT2ZDPUOM4EHZNFBZLYP54ELXK6QMKAK2XJBRRA7R6GN5EK563ZA, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.877549378 (1h44m20.658482795s) [noise=true , address=77.90.31.10:4535 , piece_id=L5O6NKB24SYXJ23URUTAFOV6YEPONAG4A2WFDXLY47N3TH2QNJKQ, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.878080592 (1h44m20.659013982s) [noise=true , address=45.91.116.9:4267 , piece_id=EY2KIBXWSQBVO73IS5WUVN3DKJ6XMKDU6KAYJRZJ7YGGZEY7M65Q, seg_pos={0 0}, upload=1598]: Uploading piece... Jul 26 23:36:04.913897176 (1h44m20.69483056s) [noise=true , address=77.90.31.13:4658 , piece_id=DBR7S55I4MPSYYU6RZ53YPY7LRGOC7NLGJTPJN5WDKKSDFKK7XVQ, seg_pos={0 0}, upload=1598]: Done uploading piece. err: ecclient: upload failed (node:129aongkQE9HMNz8vsNNA5SvKbhsHqdt9UrtSTPJJpP9mkFsboX, address:77.90.31.13:4658): protocol: storage node overloaded, request limit: 2000 Jul 26 23:36:04.966654274 (1h44m20.747587659s) [upload=1598]: Next returned nil segment Jul 26 23:36:04.966677829 (1h44m20.747611219s) [upload=1598]: Waiting for error group managing segments... Jul 26 23:36:04.974998351 (1h44m20.755931733s) [noise=true , address=77.90.31.18:4833 , piece_id=I2QO2PXWEEHCXK4AUSJRS74DI52ACO6SJZKN73TIX3DQYYQNN6PA, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:04.991088727 (1h44m20.772022089s) [noise=true , address=79.127.219.38:4010 , piece_id=TXBGG5A5OU7M4DAGIONF3JD35M2GA4CKCOWY2LE6I4TT6DXGAE4A, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:05.278501307 (1h44m21.059434686s) [noise=true , address=79.127.219.51:4196 , piece_id=U4NDAUEAXGWNBALDD32JBGNQLZQPE4T6IILT7JPWUE6WWNMYFBCQ, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:05.409195473 (1h44m21.190128838s) [noise=true , address=45.91.116.9:4267 , piece_id=EY2KIBXWSQBVO73IS5WUVN3DKJ6XMKDU6KAYJRZJ7YGGZEY7M65Q, seg_pos={0 0}, upload=1598]: Done uploading piece. err: ecclient: upload failed (node:1p6M5TgGqUwjXyM25uqv5hsh67mS81ByzMM4kWFUNaFvisDN9d, address:45.91.116.9:4267): protocol: storage node overloaded, request limit: 2000 Jul 26 23:36:05.458373441 (1h44m21.239306812s) [noise=true , address=79.127.205.235:4689 , piece_id=7OEOLJPAZX6RTIRRJW3HNBGG3CC6UCORPOGWSCBUZGM6AHA7BNXA, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:05.479084903 (1h44m21.260018282s) [noise=true , address=79.127.205.246:4281 , piece_id=PDXNFEHEAI3TSG3Z5EXHZSK4ZQ672ISPX73WSD7AP4SM55NJFPRQ, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:05.499725574 (1h44m21.280658945s) [noise=true , address=79.127.205.233:4646 , piece_id=QT2ZDPUOM4EHZNFBZLYP54ELXK6QMKAK2XJBRRA7R6GN5EK563ZA, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:05.522319306 (1h44m21.303252681s) [noise=true , address=79.127.205.245:4237 , piece_id=WGJX7TMGFWAS3ECOHZXNTNUKDXGDCYQX4RFJU2LTHHC3OE335DGQ, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:06.098427042 (1h44m21.879360425s) [noise=true , address=79.127.220.98:4421 , piece_id=FA56MS4AYJR7BDEJYF6FRIY6EPRA3IBUXKBBN6BNW6GVTIFB4JWA, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:12.552777573 (1h44m28.333710944s) [noise=true , address=45.91.116.14:4194 , piece_id=ASNDK6UDUBYSWHRCAXXEZNDOOWHAVGTY4RLADLSBKOK6FQW7GL3Q, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:12.762711647 (1h44m28.54364501s) [noise=true , address=64.49.13.140:4508 , piece_id=NUTBHS4EYXX4CTBYXQO2FVICUEO4ZONBZORHKR5M7GYBMREAINKQ, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:28.232659699 (1h44m44.013593069s) [noise=true , address=79.127.201.215:4358 , piece_id=A7C4IOLSJSOESPQH272ZD7Z4L4RRRTD73NFUPK427K35BXZQT4YQ, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:36:40.636698729 (1h44m56.41763211s) [noise=true , address=77.90.31.10:4535 , piece_id=L5O6NKB24SYXJ23URUTAFOV6YEPONAG4A2WFDXLY47N3TH2QNJKQ, seg_pos={0 0}, upload=1598]: Done uploading piece. err: <nil> Jul 26 23:37:04.754269978 (1h45m20.535203364s) [noise=true , address=104.36.82.46:4481 , piece_id=2RYU5YI4BCG5FUIDMQLOI4OAB3SJ2OPZ7OUTDISNWR3GXOW2GU5A, seg_pos={0 0}, upload=1598]: Done uploading piece. err: ecclient: upload failed (node:12bRK8WWmAnWUwzCfHu8JtZgrruYSJCvhnsYTKcaLccJcv5RQjp, address:104.36.82.46:4481): protocol: write tcp 159.223.143.39:55094->104.36.82.46:4481: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:04.754421221 (1h45m20.535354657s) [noise=true , address=104.36.82.55:4224 , piece_id=ERBTARIXECO7WA5XV3UE7G7QMAKK6PVSZAX75SB73ZH6URX6IOYA, seg_pos={0 0}, upload=1598]: Done uploading piece. err: ecclient: upload failed (node:12EjKmUiv4pxkVCB4WoiXim4uFpxrBPyfiSS4efTY3HKSE3jb5y, address:104.36.82.55:4224): protocol: rpc: dial tcp 104.36.82.55:4224: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:04.754480208 (1h45m20.535413592s) [noise=true , address=104.36.82.54:4187 , piece_id=BIWDUNNMQZGGWA7P3HOHLVEWC3LY7UUWORMY6LIOK7UN55BA55BQ, seg_pos={0 0}, upload=1598]: Done uploading piece. err: ecclient: upload failed (node:12J7goNH48bnMJyaCAPgUznrX1n6bNFNZXGqpuftNLVLxQxprsF, address:104.36.82.54:4187): protocol: rpc: dial tcp 104.36.82.54:4187: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:04.754510791 (1h45m20.535444158s) [noise=true , address=104.36.82.47:4537 , piece_id=FJZIQOODCFHS7FIVA4ZKTE2HRMPGTP5MY257VV4CYJW4Y5JV65NQ, seg_pos={0 0}, upload=1598]: Done uploading piece. err: ecclient: upload failed (node:12LTTCDNikuwf94KeKsQGFSu6ABXVYt1ombRsEyrZN5JC8d4Eot, address:104.36.82.47:4537): protocol: write tcp 159.223.143.39:50500->104.36.82.47:4537: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:37:04.754561974 (1h45m20.535495355s) [seg_pos={0 0}, upload=1598]: Done waiting for segment. successful: 12 optimal: 14 errs: <nil> Jul 26 23:37:04.787407176 (1h45m20.56834056s) [upload=1598]: Done uploading segments... Jul 26 23:37:04.787439236 (1h45m20.568372624s) [upload=1598]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 12) Duration: 1m0.035061349s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 14 out of 60 is failed (threshold=3) [.X..........XX...X...X..XX.X...X..X.......X...X....XX.......] duration: 15.133966245s FAIL (2025-07-26 23:36:54 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 15.13399621s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [X.X..........XX...X...X..XX.X...X..X.......X...X....XX......] duration: 48.392081559s FAIL (2025-07-26 23:36:39 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 48.392151508s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 38 out of 60 is failed (threshold=3) [................X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.038290865s FAIL (2025-07-26 23:36:31 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 8) Duration: 30.03830913s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 35 out of 60 is failed (threshold=3) [.........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.031293562s FAIL (2025-07-26 23:36:23 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:40800->104.36.82.9:4007: use of closed network connection; conn closed: op Jul 26 23:35:10.187069630 (1h43m25.968003081s) [seg_pos={0 0}, upload=1577]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:35:10.229261143 (1h43m26.010194523s) [upload=1577]: Done uploading segments... Jul 26 23:35:10.229291203 (1h43m26.010224581s) [upload=1577]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Jul 26 23:35:53.945803521 (1h44m9.7267369s) [upload=1595]: Starting upload Jul 26 23:35:53.945890043 (1h44m9.726823432s) [upload=1595]: Done starting upload Jul 26 23:35:53.946079246 (1h44m9.727012614s) [upload=1595]: Uploading segments... Jul 26 23:35:53.946242699 (1h44m9.727176084s) [upload=1595]: Got next segment. Inline: false Jul 26 23:35:53.946317828 (1h44m9.727251214s) [upload=1595]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:35:53.946324734 (1h44m9.727258113s) [upload=1595]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:35:54.002502015 (1h44m9.783435432s) [seg_pos={0 0}, upload=1595]: Begin upload segment... Jul 26 23:35:54.002564052 (1h44m9.783497421s) [seg_pos={0 0}, upload=1595]: Done begin upload segment. Jul 26 23:35:54.002614701 (1h44m9.783548092s) [noise=true , address=104.36.82.7:4910 , piece_id=KNJD4UFEQIZFVP3FVJS4VTFVCYDKTHWCP6AO7LM3Y4VD4BJVD7WA, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.003215323 (1h44m9.784148736s) [noise=true , address=104.36.82.15:4259 , piece_id=QONEJXNC5ONNMACLGKRWD5X74BIACXZHD7LV4U3YOAL6CN4RX3SA, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.003659788 (1h44m9.784593179s) [noise=true , address=45.91.116.11:4117 , piece_id=6EZHL64PLCD6XYMA54ASRGFKKJCNYDIIJ6V4RZIYUP2FIGRAOSGA, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.004215855 (1h44m9.785149244s) [noise=true , address=104.36.82.5:4837 , piece_id=NMFZCJDLSMSZBEV73YRPYKNKNPC4MBJM2QHFPBHI72ISCXDJDVAA, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.004606430 (1h44m9.785539816s) [noise=true , address=77.90.31.21:4879 , piece_id=ADR6FOQ4TENZFJH5KQNSLHWUSDNIK54JF2R3BSU5H76HSPNL7E4A, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.005019070 (1h44m9.785952463s) [noise=true , address=104.36.82.10:4078 , piece_id=6XMEP5XDE2425LNJA7XH6YKHOXYUMWVY3TCBYIWQYXKRW2LU24SA, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.005419168 (1h44m9.786352558s) [noise=true , address=45.91.116.7:4009 , piece_id=72GEVH5X23BLLRCARPR6XPVA4OFW24USCTUNDZ6OYRWFWGYA4KZQ, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.005753828 (1h44m9.786687206s) [noise=true , address=45.91.116.12:4227 , piece_id=2NLQHWHBYHOP5CCKSSIC46KZGNPWU4GZWETDPGYM2DQJORPQLJTQ, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.006045314 (1h44m9.786978693s) [noise=true , address=104.36.82.4:4804 , piece_id=GFZJIBUTQREFDFFCRO2U7TD3Q7UMZTZSGFGUQ33KSGXYQO7BZMPQ, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.006383534 (1h44m9.78731694s) [noise=true , address=45.91.116.8:4048 , piece_id=4XICQALXI5HDIMIQOPHN4QOATOLPV25QOPPCXFAQ2VOOZP523Y2Q, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.006788825 (1h44m9.787722207s) [noise=true , address=104.36.82.9:4007 , piece_id=F7H5YO4JZIZUE2W7FMYXAVCLBFHAXVGUHXC33BGZZRLAXDHRTCEQ, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.007315997 (1h44m9.788249396s) [noise=true , address=77.90.31.14:4661 , piece_id=DVOKWGSMWC3VQSICELUO24C2EA2ZYBB34FFO4RYD4L2PLOO5ARZA, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.007740060 (1h44m9.788673464s) [noise=true , address=77.90.31.13:4628 , piece_id=BIRTXQQST3KOOXMJL2GQM2ZJNXDK2QEG67DUPLEXHOERLOWC4IVQ, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.008127233 (1h44m9.789060639s) [noise=true , address=45.91.116.10:4083 , piece_id=6N6KEG3X3ADBK6RV34RJQXQPTLUWO752X2B3J6P2SNJ2PMR3SZNA, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.008471373 (1h44m9.789404759s) [noise=true , address=77.90.31.14:4663 , piece_id=6AW2IWP7C3MM7WJGNY45IPIGTPLKXAXO6P2FX43YZIOZWZDZZO2Q, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.008815453 (1h44m9.789748835s) [noise=true , address=45.91.116.6:4974 , piece_id=BJNYPEDFQOLVAX2Z2W7V7NBQSPLZ73K7F5G5EE7OAHUT2P6W3APA, seg_pos={0 0}, upload=1595]: Uploading piece... Jul 26 23:35:54.143934908 (1h44m9.924868276s) [upload=1595]: Next returned nil segment Jul 26 23:35:54.144039942 (1h44m9.924973323s) [upload=1595]: Waiting for error group managing segments... Jul 26 23:35:54.148543670 (1h44m9.929477041s) [noise=true , address=45.91.116.8:4048 , piece_id=4XICQALXI5HDIMIQOPHN4QOATOLPV25QOPPCXFAQ2VOOZP523Y2Q, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:35:54.149295481 (1h44m9.930228852s) [noise=true , address=45.91.116.6:4974 , piece_id=BJNYPEDFQOLVAX2Z2W7V7NBQSPLZ73K7F5G5EE7OAHUT2P6W3APA, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:35:54.149820690 (1h44m9.930754083s) [noise=true , address=45.91.116.12:4227 , piece_id=2NLQHWHBYHOP5CCKSSIC46KZGNPWU4GZWETDPGYM2DQJORPQLJTQ, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:35:54.150456218 (1h44m9.931389581s) [noise=true , address=45.91.116.10:4083 , piece_id=6N6KEG3X3ADBK6RV34RJQXQPTLUWO752X2B3J6P2SNJ2PMR3SZNA, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:35:54.151182076 (1h44m9.932115454s) [noise=true , address=77.90.31.14:4663 , piece_id=6AW2IWP7C3MM7WJGNY45IPIGTPLKXAXO6P2FX43YZIOZWZDZZO2Q, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:35:54.151697969 (1h44m9.932631351s) [noise=true , address=45.91.116.7:4009 , piece_id=72GEVH5X23BLLRCARPR6XPVA4OFW24USCTUNDZ6OYRWFWGYA4KZQ, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:35:54.152298868 (1h44m9.933232249s) [noise=true , address=77.90.31.21:4879 , piece_id=ADR6FOQ4TENZFJH5KQNSLHWUSDNIK54JF2R3BSU5H76HSPNL7E4A, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:35:54.154011055 (1h44m9.934944425s) [noise=true , address=77.90.31.14:4661 , piece_id=DVOKWGSMWC3VQSICELUO24C2EA2ZYBB34FFO4RYD4L2PLOO5ARZA, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:35:54.280558511 (1h44m10.061491902s) [noise=true , address=45.91.116.11:4117 , piece_id=6EZHL64PLCD6XYMA54ASRGFKKJCNYDIIJ6V4RZIYUP2FIGRAOSGA, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:35:55.541743144 (1h44m11.322676507s) [noise=true , address=77.90.31.13:4628 , piece_id=BIRTXQQST3KOOXMJL2GQM2ZJNXDK2QEG67DUPLEXHOERLOWC4IVQ, seg_pos={0 0}, upload=1595]: Done uploading piece. err: <nil> Jul 26 23:36:23.913398522 (1h44m39.69433191s) [noise=true , address=104.36.82.5:4837 , piece_id=NMFZCJDLSMSZBEV73YRPYKNKNPC4MBJM2QHFPBHI72ISCXDJDVAA, seg_pos={0 0}, upload=1595]: Done uploading piece. err: ecclient: upload failed (node:12rJPJdZ2EA1Uh9gUa7dHj5utLBtcFQYFPXPVX5HgJbo8ppc6pV, address:104.36.82.5:4837): protocol: rpc: dial tcp 104.36.82.5:4837: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:23.913804046 (1h44m39.694737422s) [noise=true , address=104.36.82.4:4804 , piece_id=GFZJIBUTQREFDFFCRO2U7TD3Q7UMZTZSGFGUQ33KSGXYQO7BZMPQ, seg_pos={0 0}, upload=1595]: Done uploading piece. err: ecclient: upload failed (node:12M1GKTnn8Uex751kqW5NhedsDiRb7GR7u1Dd9zwDm6XMzcRL2g, address:104.36.82.4:4804): protocol: rpc: dial tcp 104.36.82.4:4804: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:23.913915334 (1h44m39.694848725s) [noise=true , address=104.36.82.7:4910 , piece_id=KNJD4UFEQIZFVP3FVJS4VTFVCYDKTHWCP6AO7LM3Y4VD4BJVD7WA, seg_pos={0 0}, upload=1595]: Done uploading piece. err: ecclient: upload failed (node:15uiNvgWc1hPsws7NJjiH5dTx8aBh6PG4fQhsNQTqQTi9MC2Hn, address:104.36.82.7:4910): protocol: write tcp 159.223.143.39:35000->104.36.82.7:4910: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:23.914111654 (1h44m39.695045053s) [noise=true , address=104.36.82.9:4007 , piece_id=F7H5YO4JZIZUE2W7FMYXAVCLBFHAXVGUHXC33BGZZRLAXDHRTCEQ, seg_pos={0 0}, upload=1595]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: rpc: dial tcp 104.36.82.9:4007: i/o timeout; context deadline exceeded Jul 26 23:36:23.914461411 (1h44m39.695394807s) [noise=true , address=104.36.82.15:4259 , piece_id=QONEJXNC5ONNMACLGKRWD5X74BIACXZHD7LV4U3YOAL6CN4RX3SA, seg_pos={0 0}, upload=1595]: Done uploading piece. err: ecclient: upload failed (node:1ET8PaAfWq2RcPNjucMqDbmnkMZg8qnkTw9dszwLzz765oBr7y, address:104.36.82.15:4259): protocol: write tcp 159.223.143.39:42280->104.36.82.15:4259: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:23.914522870 (1h44m39.695456249s) [noise=true , address=104.36.82.10:4078 , piece_id=6XMEP5XDE2425LNJA7XH6YKHOXYUMWVY3TCBYIWQYXKRW2LU24SA, seg_pos={0 0}, upload=1595]: Done uploading piece. err: ecclient: upload failed (node:1XATjg9fmRUK8ianyYsEgDMvA3wByoKPdztMnM98XSdYBxyFdr, address:104.36.82.10:4078): protocol: write tcp 159.223.143.39:56724->104.36.82.10:4078: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:23.914569057 (1h44m39.69550244s) [seg_pos={0 0}, upload=1595]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:36:23.942316640 (1h44m39.723250007s) [upload=1595]: Done uploading segments... Jul 26 23:36:23.942360746 (1h44m39.723294114s) [upload=1595]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.031324218s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [.....................XXX.X....XX.X.X.X.......XXX...XX.....X.] duration: 793.451722ms FAIL (2025-07-26 23:36:22 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 793.511098ms edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 15 out of 60 is failed (threshold=3) [......................XXX.X....XX.X.X.X.......XXX...XX.....X] duration: 30.001191982s FAIL (2025-07-26 23:36:06 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 30.001217724s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 37 out of 60 is failed (threshold=3) [.................X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.038812307s FAIL (2025-07-26 23:36:01 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 5) Duration: 30.038923968s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.placement_20) antiblip: 49 out of 60 is failed (threshold=3) [X.XXX.XXX.X.XX....XXXX.XXXXX.X.XXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 1m0.03823959s FAIL (2025-07-26 23:36:00 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 14 but got 10) ======LOGS====== h43m16.754324307s) [noise=true , address=104.36.82.52:4110 , piece_id=24FA2HLMLASP6WPAR56VNXLVL5IDUMPB6ZZIP4TMBEJTYQ4HCNNA, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.973825790 (1h43m16.754759175s) [noise=true , address=77.90.31.19:4871 , piece_id=OFXW3YMPPV2ZIRYU56ZBIC6SBZZE4VSVDVIZ6EGYX3JLZDFZV35Q, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.974224488 (1h43m16.755157868s) [noise=true , address=79.127.205.246:4283 , piece_id=3YOJN4D4BJORXJSGKMCSZRH4LJNQTUTVLF3DXYW56DL6N7B6EXWQ, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.974555382 (1h43m16.755488761s) [noise=true , address=79.127.219.47:4048 , piece_id=FNS2XCT3IO3RARX6BVSNXHIYJG52WOO3GU3EINVIRUNE55WMWUNA, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.974906971 (1h43m16.75584035s) [noise=true , address=79.127.205.235:4666 , piece_id=F5CXC2EPUEY3TP7MNZ6GQZLEBTWRSQ2336V5ADAFV57KPXEJF72A, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.975327407 (1h43m16.75626092s) [noise=true , address=79.127.205.234:4610 , piece_id=5Q7E7MSV4U2XGOH2PBIEMWCTCATJFPO44UAASMX6KDFN6GNI2K2Q, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.975653416 (1h43m16.756586796s) [noise=true , address=104.36.82.55:4228 , piece_id=4E4ST7GBTNSANU4Z4FREQWJ4XT4GDTPLUBGJNYL2YLTE3KF6RO4A, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.976007834 (1h43m16.756941214s) [noise=true , address=104.36.82.53:4129 , piece_id=FESURDBXTPBQXCKF74GMGDXKBLKQL7PUW4KMVQMXBT2BXN2ARJIQ, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.976511755 (1h43m16.757445144s) [noise=true , address=104.36.82.47:4519 , piece_id=EV6VCGCB6BXPRVSJ6YZP2RI4ZMZ253QL2QQLBQQW7VE7K5NKWWFQ, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.976936284 (1h43m16.757869691s) [noise=true , address=79.127.219.38:4013 , piece_id=XGZ5CUTW2CEUO4KB47Q7QT4V2VL64N5FDKVTYDQGIVE4W3RD3OZQ, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.977330624 (1h43m16.75826401s) [noise=true , address=64.49.13.139:4478 , piece_id=OLE5U7SOON2TSRHXNBWXQ42WMM5K3W2KP5C5I54P2KLT3KDITCDA, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.977677177 (1h43m16.758610566s) [noise=true , address=104.36.82.56:4248 , piece_id=U4XDOZ3WJSTB4FPFAANSNPS4OOVS7U6PXHUM54Q26XOGT3XV7VTQ, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.978038406 (1h43m16.758971796s) [noise=true , address=79.127.220.101:4048 , piece_id=ZHW4564FCPGILAJNFPQ2TMFMNEU44ECWGISTRXO4PUJGI5WRCWWA, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.978415254 (1h43m16.759348637s) [noise=true , address=77.90.31.17:4784 , piece_id=6BWLTCKJKON6LCHS26WXHUZSVEYCK7FKMGQS6MAOGWPUCNAMSVVA, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.978745320 (1h43m16.7596787s) [noise=true , address=104.36.82.54:4177 , piece_id=77TVG6CBPKRWOBWUZY2OXXIK6IJRRUPLIGEWPAOK7GW4C7LCA3PA, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.979171594 (1h43m16.760104973s) [noise=true , address=104.36.82.46:4507 , piece_id=MJU7UMBGJ5MYY22S2ITFNAC36VHTLEDJDGRGM7W5BBX3NUR4DZOQ, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.979533662 (1h43m16.760467062s) [noise=true , address=77.90.31.15:4703 , piece_id=DA262R7EE6WYT4OL4IQ2IZ3OOTXY65NJRI4JIFFGKZQ4XMYVLBOA, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:00.979876289 (1h43m16.760809668s) [noise=true , address=79.127.205.245:4249 , piece_id=NB4FR2XNBFH3Z3BVBJALBFA36OKL4XSG5YOCN7IXZ5FBNMC2UVHA, seg_pos={0 0}, upload=1584]: Uploading piece... Jul 26 23:35:01.107362375 (1h43m16.888295744s) [upload=1584]: Next returned nil segment Jul 26 23:35:01.107391491 (1h43m16.888324857s) [upload=1584]: Waiting for error group managing segments... Jul 26 23:35:01.134793692 (1h43m16.915727069s) [noise=true , address=77.90.31.15:4703 , piece_id=DA262R7EE6WYT4OL4IQ2IZ3OOTXY65NJRI4JIFFGKZQ4XMYVLBOA, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:35:01.188989560 (1h43m16.969922924s) [noise=true , address=79.127.219.38:4013 , piece_id=XGZ5CUTW2CEUO4KB47Q7QT4V2VL64N5FDKVTYDQGIVE4W3RD3OZQ, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:35:01.697000738 (1h43m17.477934124s) [noise=true , address=79.127.205.234:4610 , piece_id=5Q7E7MSV4U2XGOH2PBIEMWCTCATJFPO44UAASMX6KDFN6GNI2K2Q, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:35:01.729387614 (1h43m17.510320986s) [noise=true , address=79.127.205.235:4666 , piece_id=F5CXC2EPUEY3TP7MNZ6GQZLEBTWRSQ2336V5ADAFV57KPXEJF72A, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:35:01.818316840 (1h43m17.599250222s) [noise=true , address=79.127.205.246:4283 , piece_id=3YOJN4D4BJORXJSGKMCSZRH4LJNQTUTVLF3DXYW56DL6N7B6EXWQ, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:35:01.885206499 (1h43m17.666139885s) [noise=true , address=79.127.205.245:4249 , piece_id=NB4FR2XNBFH3Z3BVBJALBFA36OKL4XSG5YOCN7IXZ5FBNMC2UVHA, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:35:02.628468909 (1h43m18.409402291s) [noise=true , address=64.49.13.139:4478 , piece_id=OLE5U7SOON2TSRHXNBWXQ42WMM5K3W2KP5C5I54P2KLT3KDITCDA, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:35:05.099767376 (1h43m20.880700741s) [noise=true , address=79.127.219.47:4048 , piece_id=FNS2XCT3IO3RARX6BVSNXHIYJG52WOO3GU3EINVIRUNE55WMWUNA, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:35:05.400170017 (1h43m21.181103382s) [noise=true , address=79.127.220.101:4048 , piece_id=ZHW4564FCPGILAJNFPQ2TMFMNEU44ECWGISTRXO4PUJGI5WRCWWA, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:35:05.877677709 (1h43m21.658611084s) [noise=true , address=77.90.31.19:4871 , piece_id=OFXW3YMPPV2ZIRYU56ZBIC6SBZZE4VSVDVIZ6EGYX3JLZDFZV35Q, seg_pos={0 0}, upload=1584]: Done uploading piece. err: <nil> Jul 26 23:36:00.810037130 (1h44m16.590970511s) [noise=true , address=104.36.82.52:4110 , piece_id=24FA2HLMLASP6WPAR56VNXLVL5IDUMPB6ZZIP4TMBEJTYQ4HCNNA, seg_pos={0 0}, upload=1584]: Done uploading piece. err: ecclient: upload failed (node:1ftGsmDR7RLZnGZzHLnE7g4PSTkpBo4JLN5MBaXPjPciPETYui, address:104.36.82.52:4110): protocol: rpc: dial tcp 104.36.82.52:4110: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:00.810135931 (1h44m16.59106931s) [noise=true , address=104.36.82.55:4228 , piece_id=4E4ST7GBTNSANU4Z4FREQWJ4XT4GDTPLUBGJNYL2YLTE3KF6RO4A, seg_pos={0 0}, upload=1584]: Done uploading piece. err: ecclient: upload failed (node:12NBsd33JtAwPdcbD8vBzZUYpjiqZpRzs8y2QrYJFE245pd8T5X, address:104.36.82.55:4228): protocol: write tcp 159.223.143.39:56682->104.36.82.55:4228: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:00.810249290 (1h44m16.591182673s) [noise=true , address=104.36.82.53:4129 , piece_id=FESURDBXTPBQXCKF74GMGDXKBLKQL7PUW4KMVQMXBT2BXN2ARJIQ, seg_pos={0 0}, upload=1584]: Done uploading piece. err: ecclient: upload failed (node:1NYQbZWuwwz7cypYmPhVwM4wygB3DsJtr3M5vxx296ZPVb3B1Z, address:104.36.82.53:4129): protocol: rpc: dial tcp 104.36.82.53:4129: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:00.810372132 (1h44m16.591305511s) [noise=true , address=104.36.82.56:4248 , piece_id=U4XDOZ3WJSTB4FPFAANSNPS4OOVS7U6PXHUM54Q26XOGT3XV7VTQ, seg_pos={0 0}, upload=1584]: Done uploading piece. err: ecclient: upload failed (node:1QpYBQ2PfiAwvt5EmFvjBnKgBhwnuyF6b77qJocpNe1YqSzP77, address:104.36.82.56:4248): protocol: rpc: dial tcp 104.36.82.56:4248: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:00.810432825 (1h44m16.591366207s) [noise=true , address=77.90.31.17:4784 , piece_id=6BWLTCKJKON6LCHS26WXHUZSVEYCK7FKMGQS6MAOGWPUCNAMSVVA, seg_pos={0 0}, upload=1584]: Done uploading piece. err: ecclient: upload failed (node:15ASyGY7W2bUF5P7q7vS7rNhxBNG24ixU4ASAsGndm8RBLMiAF, address:77.90.31.17:4784): protocol: rpc: dial tcp 77.90.31.17:4784: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:00.810495712 (1h44m16.591429095s) [noise=true , address=104.36.82.47:4519 , piece_id=EV6VCGCB6BXPRVSJ6YZP2RI4ZMZ253QL2QQLBQQW7VE7K5NKWWFQ, seg_pos={0 0}, upload=1584]: Done uploading piece. err: ecclient: upload failed (node:12enbzhEQp1y2H8Fwdb5UqxcMDdgd5HYb5jYfo2U6Fnycvscaw3, address:104.36.82.47:4519): protocol: rpc: dial tcp 104.36.82.47:4519: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:00.810502307 (1h44m16.591435687s) [noise=true , address=104.36.82.54:4177 , piece_id=77TVG6CBPKRWOBWUZY2OXXIK6IJRRUPLIGEWPAOK7GW4C7LCA3PA, seg_pos={0 0}, upload=1584]: Done uploading piece. err: ecclient: upload failed (node:1bhN3cR22QuAYihnnB3x3R97UZRWJ8sFPP8fjQcXZ7qhNZphZG, address:104.36.82.54:4177): protocol: write tcp 159.223.143.39:59922->104.36.82.54:4177: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:00.810575263 (1h44m16.591508646s) [noise=true , address=104.36.82.46:4507 , piece_id=MJU7UMBGJ5MYY22S2ITFNAC36VHTLEDJDGRGM7W5BBX3NUR4DZOQ, seg_pos={0 0}, upload=1584]: Done uploading piece. err: ecclient: upload failed (node:14jurWWPpmxwiBS3XmZysEUwugsceYJqVXFj7BbYpx8UFcTR8R, address:104.36.82.46:4507): protocol: write tcp 159.223.143.39:48250->104.36.82.46:4507: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:36:00.810607709 (1h44m16.591541099s) [seg_pos={0 0}, upload=1584]: Done waiting for segment. successful: 10 optimal: 14 errs: <nil> Jul 26 23:36:00.844247378 (1h44m16.625180761s) [upload=1584]: Done uploading segments... Jul 26 23:36:00.844303507 (1h44m16.625236887s) [upload=1584]: Upload finished. err: failed to upload enough pieces (needed at least 14 but got 10) Duration: 1m0.038260719s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20) antiblip: 15 out of 60 is failed (threshold=3) [.X.X..........XX...X...X..XX.X...X..X.......X...X....XX.....] duration: 53.960524941s FAIL (2025-07-26 23:35:50 UTC) operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 53.960549432s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uplink", Key: "orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us1.metainfo_projects.remote) antiblip: 36 out of 60 is failed (threshold=3) [..................X......XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.040356041s FAIL (2025-07-26 23:35:26 UTC) orbiter warning: uplink: failed to upload enough pieces (needed at least 13 but got 9) Duration: 30.040378536s core uplink us1 vivint
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote) antiblip: 34 out of 60 is failed (threshold=3) [..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration: 30.045336327s FAIL (2025-07-26 23:35:10 UTC) ======ERROR====== uplink: failed to upload enough pieces (needed at least 13 but got 10) ======LOGS====== .82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:34:40.057416370 (1h42m55.838349747s) [seg_pos={0 0}, upload=1569]: Done waiting for segment. successful: 12 optimal: 13 errs: <nil> Jul 26 23:34:40.082099388 (1h42m55.863032755s) [upload=1569]: Done uploading segments... Jul 26 23:34:40.082136123 (1h42m55.863069512s) [upload=1569]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 12) Jul 26 23:34:40.221455653 (1h42m56.002389043s) [upload=1577]: Starting upload Jul 26 23:34:40.221529861 (1h42m56.002463241s) [upload=1577]: Done starting upload Jul 26 23:34:40.221907241 (1h42m56.002840619s) [upload=1577]: Uploading segments... Jul 26 23:34:40.221972845 (1h42m56.002906232s) [upload=1577]: Got next segment. Inline: false Jul 26 23:34:40.222014586 (1h42m56.002947972s) [upload=1577]: Flush batch item: pb.BatchRequestItem_ObjectBegin Jul 26 23:34:40.222021086 (1h42m56.002954471s) [upload=1577]: Flush batch item: pb.BatchRequestItem_SegmentBegin Jul 26 23:34:40.252197297 (1h42m56.033130666s) [seg_pos={0 0}, upload=1577]: Begin upload segment... Jul 26 23:34:40.252246145 (1h42m56.033179509s) [seg_pos={0 0}, upload=1577]: Done begin upload segment. Jul 26 23:34:40.252280488 (1h42m56.033213855s) [noise=true , address=104.36.82.7:4912 , piece_id=T7YQQSISDNHOMQLFBBAHEQLQGSW523PR6RQQFO4QDWALMU2RUCUA, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.252805131 (1h42m56.033738511s) [noise=true , address=77.90.31.16:4735 , piece_id=D5KXRRKTLUNUEL4FJPF4MQKQURHFUTBTEEFTEIBX3CIL3PSCIJ3A, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.253198175 (1h42m56.034131559s) [noise=true , address=104.36.82.9:4007 , piece_id=GLUQRUTMINCAGR6UG7FANYL6RFTQWTP5JBEN5C5CUCYJQQOKDYQQ, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.253266603 (1h42m56.034199985s) [noise=true , address=45.91.116.14:4191 , piece_id=ODGPV6SB5JGIDBYGBZUSPNDT5DTKOVUYZPEDF6DI7ZNXEGJOJTAA, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.253599594 (1h42m56.034532985s) [noise=true , address=45.91.116.7:4009 , piece_id=XOFH5XKVBS2NPEMNHUGGGC5KOTV3THQCVNZ5XWAMK7JJL3AFKRQQ, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.253849541 (1h42m56.034782929s) [noise=true , address=45.91.116.13:4155 , piece_id=QGX3TNGBFMQ5AMYWADEAW5QN6HUM5CAGBPINAMIBLJICYPNHCVKA, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.253990096 (1h42m56.034923483s) [noise=true , address=45.91.116.8:4048 , piece_id=ODMOI2FGYQSW5X6BMPD7DU2HH7FBYTUXEABPCQLHWBSDLLFFQOUA, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.254389355 (1h42m56.035322743s) [noise=true , address=45.91.116.6:4975 , piece_id=66JBELL73G6MZX6W7MRXJJOHMMKBOCBESMJFCAG5PSCCU2MHAGRA, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.255161978 (1h42m56.036095364s) [noise=true , address=45.91.116.8:4047 , piece_id=DGCAIYY7ETLGSDK3RP4FCOTFKP4UBC5DN4OOMVO7SIE5V6QRW7DQ, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.259405366 (1h42m56.040338757s) [noise=true , address=45.91.116.12:4226 , piece_id=M3O4ZJXNS6LU7W6SGXRHDNCRLCQ2DBPQZX2W4UKJUVI6K6DZXK5A, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.259926050 (1h42m56.04085944s) [noise=true , address=104.36.82.11:4114 , piece_id=EOX3AHREXUGSRRNKRSE7VUTVULVDFL2E6OUJNDJP2L2FTJT7CFOA, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.260475877 (1h42m56.041409267s) [noise=true , address=104.36.82.13:4187 , piece_id=S2CONJADAQA3KKKF2AOECWFVVQXUL3SLCSGBQCAODKORKMTZFAEQ, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.260875663 (1h42m56.041809049s) [noise=true , address=45.91.116.11:4119 , piece_id=OEMB2DVQUTMR2WLPUSNLVX3JDJGD2PFEGHE3NERKLMECKFUFQUAQ, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.261288596 (1h42m56.042221988s) [noise=true , address=104.36.82.5:4839 , piece_id=GKTRQ5HNYVZX2UGEIMIUQZQWUYNV7UYKMTXWSMAFB65R532Z7ASQ, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.262068256 (1h42m56.043001646s) [noise=true , address=104.36.82.12:4150 , piece_id=HTEXFXTBOWURKZMALKP6WXYBPYNWCV25VENYNJ5JENMTJPQXYJRA, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.263574098 (1h42m56.044507488s) [noise=true , address=45.91.116.6:4973 , piece_id=5J3PWP6FDMMY4J23Q7UBSZ6C4QERKYAWH7N73VSMAEGLDYIPZZHQ, seg_pos={0 0}, upload=1577]: Uploading piece... Jul 26 23:34:40.363753637 (1h42m56.14468702s) [upload=1577]: Next returned nil segment Jul 26 23:34:40.363776740 (1h42m56.144710127s) [upload=1577]: Waiting for error group managing segments... Jul 26 23:34:40.369622990 (1h42m56.150556361s) [noise=true , address=45.91.116.12:4226 , piece_id=M3O4ZJXNS6LU7W6SGXRHDNCRLCQ2DBPQZX2W4UKJUVI6K6DZXK5A, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:34:40.370205840 (1h42m56.151139227s) [noise=true , address=45.91.116.8:4047 , piece_id=DGCAIYY7ETLGSDK3RP4FCOTFKP4UBC5DN4OOMVO7SIE5V6QRW7DQ, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:34:40.370231866 (1h42m56.151165304s) [noise=true , address=45.91.116.7:4009 , piece_id=XOFH5XKVBS2NPEMNHUGGGC5KOTV3THQCVNZ5XWAMK7JJL3AFKRQQ, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:34:40.370695637 (1h42m56.151629002s) [noise=true , address=45.91.116.13:4155 , piece_id=QGX3TNGBFMQ5AMYWADEAW5QN6HUM5CAGBPINAMIBLJICYPNHCVKA, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:34:40.370790748 (1h42m56.151724118s) [noise=true , address=45.91.116.8:4048 , piece_id=ODMOI2FGYQSW5X6BMPD7DU2HH7FBYTUXEABPCQLHWBSDLLFFQOUA, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:34:40.370835810 (1h42m56.15176918s) [noise=true , address=45.91.116.6:4973 , piece_id=5J3PWP6FDMMY4J23Q7UBSZ6C4QERKYAWH7N73VSMAEGLDYIPZZHQ, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:34:40.371122339 (1h42m56.152055708s) [noise=true , address=45.91.116.6:4975 , piece_id=66JBELL73G6MZX6W7MRXJJOHMMKBOCBESMJFCAG5PSCCU2MHAGRA, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:34:40.372951176 (1h42m56.15388455s) [noise=true , address=45.91.116.11:4119 , piece_id=OEMB2DVQUTMR2WLPUSNLVX3JDJGD2PFEGHE3NERKLMECKFUFQUAQ, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:34:40.409541510 (1h42m56.19047488s) [noise=true , address=77.90.31.16:4735 , piece_id=D5KXRRKTLUNUEL4FJPF4MQKQURHFUTBTEEFTEIBX3CIL3PSCIJ3A, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:34:40.552342650 (1h42m56.333276016s) [noise=true , address=45.91.116.14:4191 , piece_id=ODGPV6SB5JGIDBYGBZUSPNDT5DTKOVUYZPEDF6DI7ZNXEGJOJTAA, seg_pos={0 0}, upload=1577]: Done uploading piece. err: <nil> Jul 26 23:35:10.186470134 (1h43m25.967403516s) [noise=true , address=104.36.82.11:4114 , piece_id=EOX3AHREXUGSRRNKRSE7VUTVULVDFL2E6OUJNDJP2L2FTJT7CFOA, seg_pos={0 0}, upload=1577]: Done uploading piece. err: ecclient: upload failed (node:1rbW6vFBsSXTwat2th5XhcKj4Jmnsupc89u7KZ7zNh84JmE4JN, address:104.36.82.11:4114): protocol: rpc: dial tcp 104.36.82.11:4114: i/o timeout; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:10.186562656 (1h43m25.967496026s) [noise=true , address=104.36.82.7:4912 , piece_id=T7YQQSISDNHOMQLFBBAHEQLQGSW523PR6RQQFO4QDWALMU2RUCUA, seg_pos={0 0}, upload=1577]: Done uploading piece. err: ecclient: upload failed (node:1gwuDRBC5oJXgZbP6FqP9tc8UB6aJt2uexUgSuzHCovw27v1NR, address:104.36.82.7:4912): protocol: write tcp 159.223.143.39:33556->104.36.82.7:4912: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:10.186614272 (1h43m25.967547664s) [noise=true , address=104.36.82.13:4187 , piece_id=S2CONJADAQA3KKKF2AOECWFVVQXUL3SLCSGBQCAODKORKMTZFAEQ, seg_pos={0 0}, upload=1577]: Done uploading piece. err: ecclient: upload failed (node:12cQxBB2L2CyymYx3xWp1ALj6p8QWr4A34sqvtG5zyeT7sVWv7x, address:104.36.82.13:4187): protocol: rpc: dial tcp 104.36.82.13:4187: i/o timeout; conn closed: op; piecestore: piecestore close: context deadline exceeded Jul 26 23:35:10.186743170 (1h43m25.967676553s) [noise=true , address=104.36.82.5:4839 , piece_id=GKTRQ5HNYVZX2UGEIMIUQZQWUYNV7UYKMTXWSMAFB65R532Z7ASQ, seg_pos={0 0}, upload=1577]: Done uploading piece. err: ecclient: upload failed (node:1FcPS8yXfgdFvsxFofPrNCtFDLtebQMTaLu75335kVXeQxttuv, address:104.36.82.5:4839): protocol: write tcp 159.223.143.39:49742->104.36.82.5:4839: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:10.186864056 (1h43m25.967797426s) [noise=true , address=104.36.82.12:4150 , piece_id=HTEXFXTBOWURKZMALKP6WXYBPYNWCV25VENYNJ5JENMTJPQXYJRA, seg_pos={0 0}, upload=1577]: Done uploading piece. err: ecclient: upload failed (node:12YanYT7FkCt7SzDuqx15RP9zxaYmikVLzPkBp4i15WEShDpfpK, address:104.36.82.12:4150): protocol: write tcp 159.223.143.39:39734->104.36.82.12:4150: use of closed network connection; conn closed: op; piecestore: piecestore close: conn closed: op Jul 26 23:35:10.186952833 (1h43m25.967886198s) [noise=true , address=104.36.82.9:4007 , piece_id=GLUQRUTMINCAGR6UG7FANYL6RFTQWTP5JBEN5C5CUCYJQQOKDYQQ, seg_pos={0 0}, upload=1577]: Done uploading piece. err: ecclient: upload failed (node:19A6vk5g5ajUnXEtmXKDb9TqtjEBLYEdDvwqYsCtXKdhYUxrnp, address:104.36.82.9:4007): protocol: write tcp 159.223.143.39:40800->104.36.82.9:4007: use of closed network connection; conn closed: op Jul 26 23:35:10.187069630 (1h43m25.968003081s) [seg_pos={0 0}, upload=1577]: Done waiting for segment. successful: 10 optimal: 13 errs: <nil> Jul 26 23:35:10.229261143 (1h43m26.010194523s) [upload=1577]: Done uploading segments... Jul 26 23:35:10.229291203 (1h43m26.010224581s) [upload=1577]: Upload finished. err: failed to upload enough pieces (needed at least 13 but got 10) Duration: 30.045359687s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_us_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote) antiblip: 14 out of 60 is failed (threshold=3) [.......................XXX.X....XX.X.X.X.......XXX...XX.....] duration: 2.023487935s FAIL (2025-07-26 23:35:08 UTC) orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded Duration: 2.02351824s edge gateway us-select-1

1
...
5508
5509
5510
5511
5512
...
6004